Image processing apparatus and method

ABSTRACT

The present invention relates to an image processing apparatus and method that can generate a high-accuracy prediction image with a small amount of control information. 
     A motion compensation circuit  51  specifies a macroblock corresponding to a prediction image in a reference frame other than a current frame using a motion vector supplied from a prediction mode determination circuit  41 . The motion compensation circuit  51  reads an image of the specified macroblock from a frame memory  19 , and extracts the read image as a motion compensation image. An intra-prediction circuit  52  performs intra-prediction on the current frame using an arbitrary method to generate an intra-prediction image IP. The present invention can be applied to, for example, an encoding apparatus and a decoding apparatus.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Ser. No. 13/147,981, filedAug. 4, 2011, which is a 371 national stage of PCT/JP2010/052018 filedFeb. 12, 2010, and claims the benefit of priority under 35 U.S.C. §119from Japanese Patent Application No. 2009-036499, filed Feb. 19, 2009.The entire content of each of the foregoing applications is herebyincorporated by reference into the present application.

TECHNICAL FIELD

The present invention relates to an image processing apparatus andmethod, and more specifically to an image processing apparatus andmethod that can generate a high-accuracy prediction image withoutincreasing a processing load.

BACKGROUND ART

Conventionally, encoding schemes using motion compensation and theorthogonal transform such as the discrete cosine transform,Karhunen-Loeve transform, or wavelet transform, including MPEG (MovingPicture Experts Group), H.26x, etc., have been generally utilized asencoding schemes in the case of handling moving images. In these movingimage encoding schemes, the reduction in amount of code is achieved byutilizing the correlation in the spatial direction and time directionamong the characteristics of an input image signal to be encoded.

For example, in H.264, unidirectional prediction or bidirectionalprediction is used when an inter-frame that is a frame to be subjectedto inter-frame prediction (inter-prediction) is generated utilizing thecorrelation in the time direction. Inter-frame prediction is designed togenerate a prediction image on the basis of frames at different times.

FIG. 1 is a diagram illustrating an example of unidirectionalprediction.

As illustrated in FIG. 1, in a case where a frame to be encoded P0 thatis a frame at the present time, which is an encoding target, is to begenerated through unidirectional prediction, motion compensation isperformed using, as reference frames, encoded frames at times in thepast or future in time with respect to the present time. The residualbetween a prediction image and an actual image is encoded by utilizingthe correlation in the time direction, thus making it possible to reducethe amount of code. Reference frame information and a motion vector areused, respectively, as information specifying a reference frame andinformation specifying a position to be referred to in the referenceframe, and these pieces of information are transmitted from the encodingside to the decoding side.

Here, the number of reference frames is not limited to one. For example,in H.264, it is possible to use a plurality of frames as referenceframes. As illustrated in FIG. 1, in a case where two frames closer intime to the frame to be encoded P0 are denoted by reference frames R0and R1 in this order, a pixel value in an arbitrary macroblock in theframe to be encoded P0 can be predicted from the pixel value of anarbitrary pixel in the reference frame R0 or R1.

In FIG. 1, a box indicated inside each frame represents a macroblock. Ifa macroblock in the frame to be encoded P0, which is a predictiontarget, is represented by a macroblock MBP0, then, the macroblock in thereference frame R0 corresponding to the macroblock MBP0 is a macroblockMBR0 that is specified by a motion vector MV0. Furthermore, themacroblock in the reference frame R1 is a macroblock MBR1 that isspecified by a motion vector MV1.

If pixel values in the macroblocks MBR0 and MBR1 (pixel values in motioncompensation images) are represented by MC0(i, j) and MC1(i, j), then, apixel value in either motion compensation image is used as a pixel valuein a prediction image in unidirectional prediction. Thus, a predictionimage Pred(i, j) is represented by Equation (1) below. (i, j) representsthe relative position of a pixel in a macroblock, and satisfies 0≦i≦16and 0≦j≦16. In Equation (1), “11” indicates that one of the valuesMC0(i, j) and MC1(i, j) is taken.

[Math. 1]

Pred(i,j)=MC₀(i,j)∥MC₁(i,j)  (1)

Note that it is also possible to divide a single macroblock of 16×16pixels into sub-blocks sized by 16×8 pixels or the like and to performmotion compensation on each of the sub-blocks by referring to adifferent reference frame. Instead of motion vectors with integeraccuracy, motion vectors with decimal accuracy are transmitted andinterpolation is performed using an FIR filter defined in a standard,thus making it possible to also use the pixel values of pixels aroundthe corresponding position to be referred to for motion compensation.

FIG. 2 is a diagram illustrating an example of bidirectional prediction.

As illustrated in FIG. 2, in a case where a frame to be encoded B0 thatis a frame at the present time, which is an encoding target, is to begenerated through bidirectional prediction, motion compensation isperformed using, as reference frames, encoded frames at times in thepast and future in time with respect to the present time. The residualbetween a prediction image and an actual image is encoded by using aplurality of encoded frames as reference frames and by utilizing thecorrelation therewith, thus making it possible to reduce the amount ofcode. In H.264, it is also possible to use a plurality of frames in thepast and a plurality of frames in the future as reference frames.

As illustrated in FIG. 2, in a case where one frame in the past and oneframe in the future with respect to the frame to be encoded B0 are usedas reference frames L0 and L1, a pixel value in an arbitrary macroblockin the frame to be encoded B0 can be predicted from the pixel values ofarbitrary pixels in the reference frames L0 and L1.

In the example in FIG. 2, the macroblock in the reference frame L₀corresponding to a macroblock MBB0 in the frame to be encoded B0 is setas a macroblock MBL0 that is specified by a motion vector MV0.Furthermore, the macroblock in the reference frame L1 corresponding tothe macroblock MBB0 in the frame to be encoded B0 is set as a macroblockMBL1 that is specified by a motion vector MV1.

If pixel values of the macroblocks MBL0 and MBL1 are represented byMC0(i, j) and MC1(i, j), respectively, then, the pixel value Pred(i, j)of a prediction image Pred(i, j) can be determined as the average valueof these pixel values, as given in Equation (2) as follows.

[Math. 2]

Pred(i,j)=(MC₀(i,j)+MC₁(i,j))/2  (2)

In such motion compensation as above using unidirectional prediction,the accuracy of a prediction image is improved by increasing theaccuracy of a motion vector or by reducing the size of a macroblock, andthe residuals from the actual image are reduced, thereby achievingimprovement in encoding efficiency.

Furthermore, in motion compensation using bidirectional prediction, theaverage of the pixel values of pixels in reference frames located closein time is used as the pixel value of a pixel in a prediction image,thus making feasible a probabilistically stable reduction in predictionresidual.

FIG. 3 is a diagram illustrating an example of intra-prediction.

In the example in FIG. 3, the way prediction is performed from decodedneighboring pixels in the same screen to decode the current block of anencoded frame I0 is illustrated. In images, nearby pixel valuesgenerally have significantly high correlation. Thus, in this manner,prediction from neighboring pixels reduces residual components of thecurrent block. Thereby, improvement in encoding efficiency is realized.

For example, in intra 4×4 prediction based on the H.264 standard, it ispossible to predict the current block using nine methods by utilizingnearby encoded pixels. Two-dimensional directivity is incorporated intothe correlation with nearby images, thus realizing improvement inprediction accuracy.

As another intra-prediction method, a technique exists in which ahigh-correlation area is copied from within the screen. Specifically,the technique is such that a specific position in a decoded image isspecified in order to decode the current block and therefore thecorresponding area is utilized for a prediction image of the currentblock.

This technique provides high prediction efficiency for a regular patternor in a case where a plurality of objects having the same shape exist ina screen or in the like case.

As still another intra-prediction method, a technology also exists inwhich with the analysis of signal components in a characteristic area ora texture area existing in an encoding target image, the amount of codecan be reduced by using an artificial synthetic image for an image to beencoded.

In this manner, with the emergence of various technologies forintra-prediction, the prediction accuracy of intra-prediction has beenimproved. In general moving images, however, the prediction accuracy ofinter-prediction is still higher because, for example, even considerablycomplicated texture would provide almost zero prediction residual as aresult of inter-prediction although it is difficult to increase theaccuracy of intra-prediction in the case of stationary texture in ascreen.

Furthermore, as another prediction method, a technique has beenconsidered in which the correlation in the time direction is convertedinto the spatial resolution by motion compensation and FIR filtering ofpixel values and the spatial resolution is utilized (see, for example,NPL 1).

In the method described in NPL 1, the correlation in the time directionis utilized for the process of increasing the resolution of an inputimage sequence. Specifically, difference information on amotion-predicted/compensated image between the current image and theprevious image is calculated, and is fed back to the target currentimage to recover the high-frequency components included in the inputimage.

CITATION LIST Non Patent Literature

NPL 1: “Improving Resolution by Image Registration”, MICHAL IRANI ANDSHMUEL PELEG, Department of Computer Science, The Hebrew University ofJerusalem, 91904 Jerusalem, Israel, Communicated by Rama Chellapa,Received Jun. 16, 1989; accepted May 25, 1990

SUMMARY OF INVENTION Technical Problem

In the case of conventional inter-prediction, the use of a plurality ofreference frames requires the processing cost for motion prediction ormotion compensation or necessary cost such as the cost for the memorycapacity for saving reference planes to be higher than that in the caseof intra-prediction. This similarly applies to the method described inNPL 1.

On the other hand, in the case of intra-prediction, the processing costfor generating a prediction image is lower than that ininter-prediction; however, due to the low prediction accuracy of animage generated, there has been a problem in that the encodingefficiency is poorer than that in the case of inter-prediction.

The present invention has been made in view of the above situation, andis intended to realize improvement in encoding efficiency bycompensating for the lack of prediction accuracy of intra-prediction byusing the prediction accuracy of inter-prediction and to realize areduction in processing cost by reducing the number of reference planesnecessary for inter-prediction.

Solution to Problem

One aspect of the present invention provides an image processingapparatus including decoding means for decoding an encoded image;generating means for adding together the image decoded by the decodingmeans and a prediction image, and for generating a decoded image;extracting means for performing motion compensation using, as areference frame, a frame formed of the decoded image generated by thegenerating means and using a motion vector in the encoded image, and forextracting a motion compensation image corresponding to the predictionimage from the reference frame; intra-prediction image generating meansfor performing intra-frame prediction for a present frame for which theprediction image is to be generated, and for generating anintra-prediction image corresponding to the prediction image from aportion of the decoded image generated by the generating means; andprediction image generating means for generating the prediction image byperforming a filtering process for compensating the motion compensationimage extracted by the extracting means and the intra-prediction imagegenerated by the intra-prediction image generating means forhigh-frequency components by utilizing correlation in a time directionthat is included in the motion compensation image and theintra-prediction image.

The prediction image generating means can include first filter means forapplying a low-pass filter to a difference image between the motioncompensation image extracted by the first extracting means and theintra-prediction image generated by the intra-prediction imagegenerating means; second filter means for applying a high-pass filter toan image obtained by applying a low-pass filter by using the firstfilter means; and adding means for adding the image obtained by applyinga low-pass filter by using the first filter means and an image obtainedby applying a high-pass filter by using the second filter means to theintra-prediction image generated by the intra-prediction imagegenerating means, and for generating the prediction image.

The adding means can add the image obtained by applying a low-passfilter by using the first filter means and the image obtained byapplying a high-pass filter by using the second filter means to a motioncompensation image extracted from a preceding frame with respect to atime of the prediction image.

The image processing apparatus can further include unidirectionalprediction means for performing unidirectional prediction using aplurality of motion compensation images, and for generating theprediction image; bidirectional prediction means for performingbidirectional prediction using a plurality of motion compensationimages, and for generating the prediction image; and judging means forjudging whether the prediction image is to be generated throughunidirectional prediction by the unidirectional prediction means,generated through bidirectional prediction by the bidirectionalprediction means, or generated through the filtering process by theprediction image generating means, by using an identification flagincluded in a header of the encoded image.

One aspect of the present invention further provides an image processingmethod including decoding an encoded image; adding together an imagethat has been decoded and a prediction image and generating a decodedimage; performing motion compensation using, as a reference frame, aframe formed of the generated decoded image and using a motion vector inthe encoded image, and extracting a motion compensation imagecorresponding to the prediction image from the reference frame;performing intra-frame prediction for a present frame for which theprediction image is to be generated, and generating an intra-predictionimage corresponding to the prediction image from a portion of thedecoded image; and generating the prediction image by performing afiltering process for compensating the motion compensation image and theintra-prediction image for high-frequency components by utilizingcorrelation in a time direction that is included in the motioncompensation image and the intra-prediction image.

Another aspect of the present invention provides an image processingapparatus including encoding means for encoding an original image thatis an encoding target image, and for generating an encoded image;detecting means for detecting a motion vector on the basis of an imageand the original image, the image being obtained by performing localdecoding on the basis of a residual signal indicating a differencebetween the original image and a prediction image; extracting means forperforming motion compensation using, as a reference frame, a frameformed of the image obtained by performing local decoding and using themotion vector detected by the detecting means, and for extracting amotion compensation image corresponding to the prediction image from thereference frame; intra-prediction image generating means for performingintra-frame prediction for a present frame for which the predictionimage is to be generated, and for generating an intra-prediction imagecorresponding to the prediction image from a portion of an image of theframe; and generating means for generating the prediction image byperforming a filtering process for compensating the motion compensationimage extracted by the first extracting means and the intra-predictionimage generated by the intra-prediction image generating means forhigh-frequency components by utilizing correlation in a time directionthat is included in the motion compensation image.

The generating means can include first filter means for applying alow-pass filter to a difference image between the motion compensationimage extracted by the first extracting means and the intra-predictionimage generated by the intra-prediction image generating means; secondfilter means for applying a high-pass filter to an image obtained byapplying a low-pass filter by using the first filter means; and addingmeans for adding the image obtained by applying a low-pass filter byusing the first filter means and an image obtained by applying ahigh-pass filter by using the second filter means to theintra-prediction image generated by the intra-prediction imagegenerating means, and for generating the prediction image.

The adding means can add the image obtained by applying a low-passfilter by using the first filter means and the image obtained byapplying a high-pass filter by using the second filter means to a motioncompensation image extracted from a preceding frame with respect to atime of the prediction image.

The encoding means can incorporate an identification flag into a headerof the encoded image, the identification flag identifying whether aprediction image to be added to an image decoded by a decoding apparatusis to be generated through unidirectional prediction, generated throughbidirectional prediction, or generated through the filtering process.

Another aspect of the present invention further provides an imageprocessing method including encoding an original image that is anencoding target image, and generating an encoded image; detecting amotion vector on the basis of an image and the original image, the imagebeing obtained by performing local decoding on the basis of a residualsignal indicating a difference between the original image and aprediction image; performing motion compensation using, as a referenceframe, a frame formed of the image obtained by performing local decodingand using the detected motion vector, and extracting a motioncompensation image corresponding to the prediction image from thereference frame; performing intra-frame prediction for a present framefor which the prediction image is to be generated, and generating anintra-prediction image corresponding to the prediction image from aportion of an image of the frame; and generating the prediction image byperforming a filtering process for compensating the extracted motioncompensation image and the generated intra-prediction image forhigh-frequency components by utilizing correlation in a time directionthat is included in the motion compensation image.

In one aspect of the present invention, an encoded image is decoded; animage that has been decoded and a prediction image are added together; adecoded image is generated; motion compensation is performed using, as areference frame, a frame formed of the generated decoded image and usinga motion vector in the encoded image; a motion compensation imagecorresponding to the prediction image is extracted from the referenceframe; intra-frame prediction is performed for a present frame for whichthe prediction image is to be generated; an intra-prediction imagecorresponding to the prediction image is generated from a portion of thedecoded image; and a filtering process for compensating the motioncompensation image and the intra-prediction image for high-frequencycomponents by utilizing correlation in a time direction that is includedin the motion compensation image and the intra-prediction image isperformed, thereby generating the prediction image.

In another aspect of the present invention, an original image that is anencoding target image is encoded; an encoded image is generated; amotion vector is detected on the basis of an image and the originalimage, the image being obtained by performing local decoding on thebasis of a residual signal indicating a difference between the originalimage and a prediction image; motion compensation is performed using, asa reference frame, a frame formed of the image obtained by performinglocal decoding and using the detected motion vector; a motioncompensation image corresponding to the prediction image is extractedfrom the reference frame; intra-frame prediction is performed for apresent frame for which the prediction image is to be generated; anintra-prediction image corresponding to the prediction image isgenerated from a portion of an image of the frame; and a filteringprocess for compensating the extracted motion compensation image and thegenerated intra-prediction image for high-frequency components byutilizing correlation in a time direction that is included in the motioncompensation image is performed, thereby generating the predictionimage.

Advantageous Effects of Invention

According to the present invention, a high-accuracy prediction image canbe generated and high encoding efficiency can be achieved withoutincreasing the amount of transmission of motion vectors in a stream.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of unidirectionalprediction.

FIG. 2 is a diagram illustrating an example of bidirectional prediction.

FIG. 3 is a diagram illustrating an example of intra-prediction.

FIG. 4 is a diagram explaining an overview of the generation of aprediction image in the present invention.

FIG. 5 is a block diagram illustrating an example configuration of adecoding apparatus according to an embodiment of the present invention.

FIG. 6 is a diagram illustrating a concept of a third prediction mode.

FIG. 7 is a diagram illustrating a concept of the third prediction mode.

FIG. 8 is a block diagram illustrating an example configuration of amotion prediction/compensation circuit in FIG. 5.

FIG. 9 is a diagram illustrating an example of reference frames.

FIG. 10 is a diagram illustrating another example of reference frames.

FIG. 11 is a block diagram illustrating an example configuration of aprediction circuit in FIG. 8.

FIG. 12 is a block diagram illustrating an example configuration of afiltering circuit in FIG. 8.

FIG. 13 is a flowchart explaining a decoding process of the decodingapparatus.

FIG. 14 is a flowchart explaining a motion prediction/compensationprocess performed in step S9 in FIG. 13.

FIG. 15 is a flowchart explaining an example of a flow of an extractionprocess.

FIG. 16 is a flowchart explaining an example of a flow of a filteringprediction process.

FIG. 17 is a block diagram illustrating an example configuration of anencoding apparatus.

FIG. 18 is a block diagram illustrating an example configuration of amode determination circuit in FIG. 17.

FIG. 19 is a block diagram illustrating an example configuration of amotion prediction/compensation circuit in FIG. 17.

FIG. 20 is a flowchart explaining an encoding process of the encodingapparatus.

FIG. 21 is a flowchart explaining a mode determination process performedin step S108 in FIG. 20.

FIG. 22 is a flowchart explaining a motion prediction/compensationprocess performed in step S111 in FIG. 20.

FIG. 23 is a block diagram illustrating another example configuration ofthe filtering circuit.

FIG. 24 is a block diagram illustrating still another exampleconfiguration of the filtering circuit.

FIG. 25 is a diagram illustrating an example in a case where threereference frames are used.

FIG. 26 is a block diagram illustrating an example configuration of thefiltering circuit in a case where three reference frames are used.

FIG. 27 is a block diagram illustrating an example configuration of apersonal computer.

FIG. 28 is a block diagram illustrating an example configuration of amain part of a television receiver to which the present invention isapplied.

FIG. 29 is a block diagram illustrating an example configuration of amain part of a mobile phone to which the present invention is applied.

FIG. 30 is a block diagram illustrating an example configuration of amain part of a hard disk recorder to which the present invention isapplied.

FIG. 31 is a block diagram illustrating an example configuration of amain part of a camera to which the present invention is applied.

FIG. 32 is a diagram illustrating examples of the size of a macroblock.

DESCRIPTION OF EMBODIMENTS

Modes for carrying out the invention (hereinafter, embodiments) will beexplained hereinafter. Note that the explanation will be given in thefollowing order:

1. First embodiment (decoding process)2. Second embodiment (encoding process)3. Third embodiment (modifications of filtering circuit)

1. First Embodiment Overview of Prediction

FIG. 4 is a diagram explaining an overview of a prediction imagegeneration method to which the present invention is applied. In thepresent invention, in a decoder, at least one motion vector istransmitted through a bit stream in order to obtain a motioncompensation image.

In FIG. 4, the way a single frame (N−1) is used for a reference planefor use in motion compensation in order to decode a frame N isillustrated. In FIG. 4, a motion vector for indicating the coordinatesof an image to be subjected to motion compensation for the frame (N−1)is transmitted by a stream. The decoder uses this vector to obtain animage MC.

Furthermore, intra-prediction is performed utilizing a decoded pixelvalue in the frame N. In this prediction, for example, intra-predictionbased on the H.264 standard is used. However, the present invention doesnot limit the type of intra-prediction process, and any predictionsuitable for higher-accuracy prediction using the subsequent filteringprocess by utilizing an encoded pixel in the current frame may beselected.

With the above two prediction processes, the decoder can obtain a motionprediction image from the frame (N−1) and a spatial prediction imagefrom the frame N when decoding the current block. The two types ofimages are subjected to a filtering process to generate a new predictionimage closer to the original image by utilizing components included inan input prediction image.

[Configuration of Decoding Apparatus]

FIG. 5 is a block diagram illustrating an example configuration of adecoding apparatus 1 according to an embodiment of the presentinvention.

Image information encoded by an encoding apparatus described below isinput to the decoding apparatus 1 via a cable, a network, or a removablemedium. Examples of the compressed image information include imageinformation encoded in accordance with the H.264 standard.

A storage buffer 11 sequentially stores bit streams input as compressedimage information. The information stored in the storage buffer 11 isread by a lossless decoding circuit 12 in units of images of certainunits such as macroblocks forming a frame, as appropriate. In the H.264standard, it is also possible to perform processing, instead of in unitsof macroblocks of 16×16 pixels, in units of blocks into which amacroblock is further divided, such as blocks of 8×8 pixels or 4×4pixels.

The lossless decoding circuit 12 performs a decoding processcorresponding to an encoding scheme, such as a variable-length decodingprocess or an arithmetic decoding process, on an image read from thestorage buffer 11. The lossless decoding circuit 12 outputs a quantizedtransform coefficient, which is obtained by performing the decodingprocess, to a dequantization circuit 13.

Furthermore, the lossless decoding circuit 12 identifies, on the basisof an identification flag included in the header of the image to bedecoded, whether this image is an intra-coded image or an inter-codedimage. In a case where it is judged that the image to be decoded is anintra-coded image, the lossless decoding circuit 12 outputsintra-prediction mode information stored in the header of the image toan intra-prediction circuit 22. The intra-prediction mode informationincludes information regarding intra-prediction, such as the size of ablock used as a unit of processing.

In a case where it is judged that the image to be decoded is inter-codedinformation, the lossless decoding circuit 12 outputs motion vectors andan identification flag that are stored in the header of the image to amotion prediction/compensation circuit 21. The mode of prediction inwhich a prediction image is to be generated through inter-prediction isidentified using the identification flag. Identification flags are setin units of, for example, macroblocks or frames.

In addition to the mode of unidirectional prediction in FIG. 1 and themode of bidirectional prediction in FIG. 2, the modes of prediction thatare prepared include a third prediction mode for generating a predictionimage by performing filtering on motion compensation images extractedfrom a plurality of reference frames located in one or both temporaldirections.

FIG. 6 is a diagram illustrating an concept of the third predictionmode.

In the example in FIG. 6, the temporally preceding frame with respect tothe time of the present frame (prediction frame) is set as a referenceframe R0, and the preceding frame with respect to the reference frame R0is set as a reference frame R1. In this case, according to the thirdprediction mode, motion compensation images MC0 and MC1 extracted fromthe reference frames R0 and R1 are input to a filtering circuit, and apixel value in an image output from the filtering circuit is set as apixel value in a prediction image that is the target macroblock.

FIG. 7 is a diagram illustrating a concept in a case where differentimages are input in the third prediction mode.

In the example in FIG. 7, the temporally preceding frame with respect tothe time of the present frame (prediction frame) is set as a referenceframe R0. In this case, for the sake of the third prediction mode, amotion compensation image MC0 extracted from the reference frame R0 andan intra-prediction image IP generated from an encoded image that islocated near the current block in the present frame are input to afiltering circuit, and a pixel value in an image output from thefiltering circuit is set as a pixel value in a prediction image that isthe target macroblock.

Hereinafter, the mode of prediction, as explained with reference to FIG.1, in which a pixel value in one motion compensation image among motioncompensation images extracted from a plurality of reference frameslocated in one direction is set as a pixel value in a prediction imageis simply referred to as a unidirectional prediction mode. Furthermore,the mode of prediction, as explained with reference to FIG. 2, in whichthe average value of pixel values in motion compensation imagesindividually extracted from a plurality of reference frames located inboth directions is set as a pixel value in a prediction image is simplyreferred to as a bidirectional prediction mode.

The third mode of prediction, as illustrated in FIG. 6, in which a pixelvalue in a prediction image is determined by performing filtering oneach of motion compensation images extracted from a plurality ofreference frames located in one or both directions is referred to as afiltering prediction mode.

Similarly, the third mode of prediction, as illustrated in FIG. 7, inwhich a pixel value in a prediction image is determined by performingfiltering on, in addition to each motion compensation image extractedfrom one or a plurality of reference frames, an intra-prediction imageextracted by performing intra-prediction from an encoded image in thepresent frame is also referred to as a filtering prediction mode. Thisfiltering prediction mode will be described in detail below.

Referring back to FIG. 5, the dequantization circuit 13 performsdequantization on the quantized transform coefficient supplied from thelossless decoding circuit 12 using a scheme corresponding to thequantization scheme used on the encoding side. The dequantizationcircuit 13 outputs a transform coefficient obtained by performingdequantization to an inverse orthogonal transform circuit 14.

The inverse orthogonal transform circuit 14 performs, for example, thefourth-order inverse orthogonal transform on the transform coefficientsupplied from the dequantization circuit 13 using a scheme correspondingto the orthogonal transform scheme used on the encoding side, such asthe discrete cosine transform or Karhunen-Loeve transform, and outputsan obtained image to an adder circuit 15.

The adder circuit 15 combines the decoded image supplied from theinverse orthogonal transform circuit 14 and a prediction image suppliedfrom the motion prediction/compensation circuit 21 or from theintra-prediction circuit 22 via a switch 23, and outputs a compositeimage to a deblocking filter 16.

The deblocking filter 16 removes block noise included in the imagesupplied from the adder circuit 15, and outputs an image from which theblock noise has been removed. The image output from the deblockingfilter 16 is supplied to a rearrangement buffer 17 and a frame memory19.

The rearrangement buffer 17 temporarily stores the image supplied fromthe deblocking filter 16. The rearrangement buffer 17 generates anindividual frame from an image of, for example, every macroblock that isstored, and rearranges the generated frames in certain order such asdisplay order before outputting them to a D/A (Digital/Analog) convertercircuit 18.

The D/A converter circuit 18 performs D/A conversion on each of theframes supplied from the rearrangement buffer 17, and outputs thesignals of the frames to outside.

The frame memory 19 temporarily stores the image supplied from thedeblocking filter 16. The information stored in the frame memory 19 issupplied to the motion prediction/compensation circuit 21 or theintra-prediction circuit 22 via a switch 20.

The switch 20 is connected to a terminal a1 in a case where a predictionimage is to be generated through inter-prediction, and is connected to aterminal b1 in a case where a prediction image is to be generatedthrough intra-prediction. The switching of the switch 20 is controlledby, for example, a control circuit 31.

The motion prediction/compensation circuit 21 determines a predictionmode in accordance with the identification flag supplied from thelossless decoding circuit 12, and selects a frame to be used as areference frame, from among decoded frames that are stored in the framememory 19, in accordance with the prediction mode. The motionprediction/compensation circuit 21 determines a macroblock correspondingto the target prediction image from among macroblocks forming thereference frame on the basis of the motion vectors supplied from thelossless decoding circuit 12, and extracts the determined macroblock asa motion compensation image. The motion prediction/compensation circuit21 determines a pixel value in a prediction image from a pixel value inthe motion compensation image in accordance with the prediction mode,and outputs the prediction image whose pixel values have been determinedto the adder circuit 15 via the switch 23.

The intra-prediction circuit 22 performs intra-prediction in accordancewith the intra-prediction mode information supplied from the losslessdecoding circuit 12, and generates a prediction image. Theintra-prediction circuit 22 outputs the generated prediction image tothe adder circuit 15 via the switch 23.

The switch 23 is connected to a terminal a2 in a case where a predictionimage has been generated by the motion prediction/compensation circuit21, and is connected to a terminal b2 in a case where a prediction imagehas been generated by the intra-prediction circuit 22. The switching ofthe switch 23 is also controlled by, for example, the control circuit31.

The control circuit 31 controls the overall operation of the decodingapparatus 1 by, for example, switching the connection of the switches 20and 23. The identification of whether the processing target image is anintra-coded image or an inter-coded image may be performed by thecontrol circuit 31.

FIG. 8 is a block diagram illustrating an example configuration of themotion prediction/compensation circuit 21 in FIG. 5.

As illustrated in FIG. 8, the motion prediction/compensation circuit 21is composed of a prediction mode determination circuit 41, aunidirectional prediction circuit 42, a bidirectional prediction circuit43, a prediction circuit 44, and a filtering circuit 45. The motionvectors and identification flag supplied from the lossless decodingcircuit 12 are input to the prediction mode determination circuit 41.

The prediction mode determination circuit 41 determines a predictionmode in accordance with the identification flag supplied from thelossless decoding circuit 12. The prediction mode determination circuit41 outputs the motion vectors to the unidirectional prediction circuit42 in a case where it is determined that a prediction image is to begenerated through unidirectional prediction, and outputs the motionvectors to the bidirectional prediction circuit 43 in a case where it isdetermined that a prediction image is to be generated throughbidirectional prediction. Furthermore, in a case where it is determinedthat a prediction image is to be generated through filtering prediction,the prediction mode determination circuit 41 outputs the motion vectorsto the prediction circuit 44.

In this manner, a value different from a value representingunidirectional prediction and a value representing bidirectionalprediction, which are defined in the conventional H.264 standard, can beset as the value of an identification flag, in order to enableidentification of filtering prediction. Note that a prediction mode maybe determined using a method determined in advance, instead of beingdetermined in accordance with an identification flag, to reduce theamount of information.

The unidirectional prediction circuit 42 sets, as illustrated in FIG. 1,a plurality of frames located in one temporal direction as referenceframes, and determines macroblocks in the reference frames correspondingto a prediction image on the basis of motion vectors. Furthermore, theunidirectional prediction circuit 42 generates a prediction image byreading each of the determined macroblocks in the reference frames fromthe frame memory 19 as a motion compensation image and by setting apixel value in one of the motion compensation images as a pixel value inthe prediction image. The unidirectional prediction circuit 42 outputsthe prediction image to the adder circuit 15. For example,unidirectional prediction defined in the H.264 standard is used as theunidirectional prediction performed by the unidirectional predictioncircuit 42.

The bidirectional prediction circuit 43 sets, as illustrated in FIG. 2,a plurality of frames located in temporal both directions as referenceframes, and determines macroblocks in the reference frames correspondingto a prediction image on the basis of motion vectors. Furthermore, thebidirectional prediction circuit 43 generates a prediction image byreading each of the determined macroblocks in the reference frames fromthe frame memory 19 as a motion compensation image and by setting theaverages of pixel values in the read motion compensation images as apixel value in the prediction image. The bidirectional predictioncircuit 43 outputs the prediction image to the adder circuit 15. Forexample, bidirectional prediction defined in the H.264 standard is usedas the bidirectional prediction performed by the bidirectionalprediction circuit 43.

The prediction circuit 44 determines a plurality of frames located inone or both temporal directions as reference frames. Which frames are tobe used as reference frames may be determined in advance or may bespecified by information transmitted from the encoding side togetherwith an identification flag.

FIG. 9 is a diagram illustrating an example of reference frames.

In the example in FIG. 9, the temporally preceding frame with respect tothe time of a prediction frame and the further preceding frame, i.e.,two frames, are set as reference frames. Of the two reference frames,the preceding frame closer to the prediction frame is set as a referenceframe R0, and the preceding frame with respect to the reference frame R0is set as a reference frame R1.

FIG. 10 is a diagram illustrating another example of reference frames.

In the example in FIG. 10, the temporally preceding and following frameswith respect to the time of a prediction frame, i.e., two frames, areset as reference frames. Of the two reference frames, the precedingframe with respect to the prediction frame is set as a reference frameL0, and the following frame is set as a reference frame L1.

In this manner, in filtering prediction, a plurality of frames locatedin one temporal direction or a plurality of frames located in bothdirections are used as reference frames.

Furthermore, the prediction circuit 44 determines a macroblockcorresponding to the prediction image, among decoded macroblocks in atleast one reference frame among the reference frames determined in themanner as illustrated in FIG. 9 or 10, on the basis of the motionvectors supplied from the prediction mode determination circuit 41.

Moreover, as explained with reference to FIG. 7, the prediction circuit44 performs intra-prediction on the current frame in accordance with,for example, the H.264 standard.

The prediction circuit 44 reads the determined macroblock in each of thereference frames from the frame memory 19 as an intra-prediction image,and outputs the read intra-prediction images to the filtering circuit45.

That is to say, the prediction circuit 44 extracts a motion compensationimage on the basis of a motion vector from a reference frame other thanthe current frame, and also generates an intra-prediction image byperforming intra-prediction on the current frame.

Note that motion vectors may be determined, instead of in units ofmacroblocks of 16×16 pixels or the like, in units of blocks into which amacroblock is further divided. An image of, for example, everymacroblock is input to the filtering circuit 45. In FIG. 8, therepresentation of two arrows as arrows directed from the predictioncircuit 44 to the filtering circuit 45 indicates that two motioncompensation images are supplied.

The filtering circuit 45 receives, as inputs, the motion compensationimages supplied from the prediction circuit 44, performs filtering, andoutputs a prediction image obtained by performing filtering to the addercircuit 15.

FIG. 11 is a block diagram illustrating an example configuration of theprediction circuit 44 in FIG. 6. In FIG. 11, the prediction circuit 44includes a motion compensation circuit 51 and an intra-predictioncircuit 52.

The motion compensation circuit 51 specifies a macroblock correspondingto the prediction image in a reference frame other than the currentframe using the motion vectors supplied from the prediction modedetermination circuit 41. The motion compensation circuit 51 reads theimage of the specified macroblock from the frame memory 19, and extractsthe read image as a motion compensation image. The motion compensationcircuit 51 supplies the extracted motion compensation image MC0 to thefiltering circuit 45.

The intra-prediction circuit 52 performs intra-prediction (intra-frameprediction) on the current frame (present frame) using an arbitrarymethod, and generates an intra-prediction image IP corresponding to thedesired prediction image as the prediction result. Here, in order toperform intra-prediction, for example, H.264 intra 4×4 prediction, intra8×8 prediction, or intra 16×16 prediction is used. In a case where theabove intra-prediction is used, control information for uniquelydefining a prediction method on the decoder side is described in astream and is transmitted. The intra-prediction circuit 52 supplies thegenerated intra-prediction image IP to the filtering circuit 45.

FIG. 12 is a block diagram illustrating an example configuration of thefiltering circuit 45. In the filtering circuit 45 having theconfiguration in FIG. 12, filtering is performed on time domain signals.

As illustrated in FIG. 12, the filtering circuit 45 is composed of adifference calculation circuit 61, a low-pass filter circuit 62, a gainadjustment circuit 63, a high-pass filter circuit 64, a gain adjustmentcircuit 65, an adder circuit 66, and an adder circuit 67. The motioncompensation image MC0 supplied from the prediction circuit 44 is inputto the difference calculation circuit 61 and the adder circuit 67, andthe motion compensation image MC1 is input to the difference calculationcircuit 61.

As described above, in a case where a prediction image is generatedthrough intra-prediction, for example, an image generated in the currentframe is set as the intra-prediction image IP, and an image extractedfrom a reference frame other than the current frame is set as the motioncompensation image MC1.

Note that in a case where a prediction image is generated throughunidirectional prediction in the manner as illustrated in FIG. 9, forexample, an image extracted from the reference frame R0, which can beconsidered to have higher correlation with the prediction image, is setas the motion compensation image MC0, and an image extracted from thereference frame R1 is set as the motion compensation image MC1. Theimage extracted from the reference frame R0 may be designed to be set asthe motion compensation image MC1, and the image extracted from thereference frame R1 may be designed to be set as the motion compensationimage MC0.

Furthermore, in a case where a prediction image is generated throughbidirectional prediction in the manner as illustrated in FIG. 10, forexample, an image extracted from the preceding reference frame L0 is setas the motion compensation image MC0, and an image extracted from thefollowing reference frame L1 is set as the motion compensation imageMC1. The image extracted from the reference frame L0 may be designed tobe set as the motion compensation image MC1, and the image extractedfrom the reference frame L1 may be designed to be set as the motioncompensation image MC0.

The above motion compensation image MC0 can be replaced by theintra-prediction image IP in FIG. 12, and can be processed in a mannersimilar to that of the intra-prediction image IP. The followingexplanation will be given of the intra-prediction image IP.

The difference calculation circuit 61 calculates the difference betweenthe intra-prediction image IP (motion compensation image MC0) and themotion compensation image MC1, and outputs a difference image to thelow-pass filter circuit 62. The difference image D is represented byEquation (3) as follows:

[Math. 3]

D(i,j)=IP(i,j)−MC₁(i,j)  (3)

In Equation (3), (i, j) represents the relative position of a pixel in amotion compensation image, and satisfies 0≦i≦16 and 0≦j≦16 in a casewhere the process is designed to be performed in units of macroblocks of16×16 pixels. It is assumed that this similarly applies to the followingdescription.

The low-pass filter circuit 62 has an FIR filter circuit. The low-passfilter circuit 62 applies a low-pass filter to the difference image Dsupplied from the difference calculation circuit 61, and outputs anobtained image to the gain adjustment circuit 63 and the high-passfilter circuit 64. A difference image D′, which is an image obtained byapplying a low-pass filter, is represented by Equation (4) below. InEquation (4), LPF(X) represents the application of a low-pass filter toan input image X using a two-dimensional FIR filter.

[Math. 4]

D′=LPF(D)  (4)

The gain adjustment circuit 63 adjusts the gain of the difference imageD′ supplied from the low-pass filter circuit 62, and outputs thegain-adjusted image to the adder circuit 66. The output image X(i, j) ofthe gain adjustment circuit 63 is represented by Equation (5) asfollows.

[Math. 5]

X(i,j)=αD′(i,j)  (5)

The high-pass filter circuit 64 has an FIR filter circuit. The high-passfilter circuit 64 applies a high-pass filter to the difference image D′supplied from the low-pass filter circuit 62, and outputs an obtainedimage to the gain adjustment circuit 65. A difference image D″, which isan image obtained by applying a high-pass filter, is represented byEquation (6) below. In Equation (6), HPF(X) represents the applicationof a high-pass filter to an input image X using a two-dimensional FIRfilter.

[Math. 6]

D″=HPF(D′)  (6)

The gain adjustment circuit 65 adjusts the gain of the difference imageD″ supplied from the high-pass filter circuit 64, and outputs thegain-adjusted image to the adder circuit 66. The output image Y(i, j) ofthe gain adjustment circuit 65 is represented by Equation (7) asfollows.

[Math. 7]

Y(i,j)=βD″(i,j)  (7)

As the values of a in Equation (5) and β in Equation (7), for example,values such as α=0.8 and β=0.2 are selected. However, other values maybe used in order to increase the accuracy of a prediction pixel.Furthermore, the values may be adaptively changed in accordance with theproperties of an input sequence or the like.

The adder circuit 66 adds together the gain-adjusted image X(i, j) andimage Y(i, j), and outputs an image obtained by addition. The outputimage Z(i, j) of the adder circuit 66 is represented by Equation (8) asfollows.

[Math. 8]

Z(i,j)=X(i,j)+Y(i,j)  (8)

The output image Z(i, j) is a representation of the high-frequencycomponents of the image, which can be determined from the difference,that is, the correlation, between the motion compensation image MC0 andthe motion compensation image MC1.

The adder circuit 67 adds the output image Z(i, j) supplied from theadder circuit 66 to the intra-prediction image IP, and outputs anobtained image to the adder circuit 15 as a prediction image. The finaloutput of the adder circuit 67, i.e., a prediction image S(i, j), isrepresented by Equation (9) as follows.

[Math. 9]

S(i,j)=IP(i,j)+Z(i,j)  (9)

In this manner, according to the filtering prediction mode, an imageobtained by adding an image representing high-frequency components tothe intra-prediction image IP is generated as a prediction image. Thisprediction image includes more high-frequency components than aprediction image obtained in a case where bidirectional prediction issimply performed. Further, it is possible to generate a prediction imageby more efficiently utilizing the temporal correlation of images than ina case where unidirectional prediction is simply performed. Furthermore,since a prediction image including a large number of high-frequencycomponents is added with a decoded image in the adder circuit 15, animage that is finally output from the decoding apparatus 1 is also ahigh-definition image including a large number of high-frequencycomponents.

Furthermore, in the generation of a prediction image, intra-predictionis performed on the current frame and the thus generatedintra-prediction image IP is utilized together with the motioncompensation image MC1. This can reduce the number of motion vectors forgenerating motion compensation images.

As discussed above, after a motion compensation apparatus according tothe present invention outputs a prediction image, image information isrecovered by performing decoding in a conventional manner.

In this manner, the decoding apparatus 1 can realize improvement inencoding efficiency by compensating for the lack of prediction accuracyof intra-prediction by using the prediction accuracy ofinter-prediction, and can also realize a reduction in processing cost byreducing the number of reference planes necessary for inter-prediction.

[Explanations of Flow of Decoding Process]

Here, a process of the decoding apparatus 1 having the aboveconfiguration will be explained.

First, a decoding process of the decoding apparatus 1 will be explainedwith reference to a flowchart in FIG. 13.

The process in FIG. 13 is started when, for example, an image of acertain size such as a macroblock of 16×16 pixels is read by thelossless decoding circuit 12 from the information stored in the storagebuffer 11. The processing of each step in FIG. 13 is performed, asappropriate, in parallel with the processing of another step or byreordering the steps. This similarly applies to the processing of eachstep in each flowchart described below.

In step S1, the lossless decoding circuit 12 performs a decoding processon an image read from the storage buffer 11, and outputs a quantizedtransform coefficient to the dequantization circuit 13. Furthermore, thelossless decoding circuit 12 outputs intra-prediction mode informationto the intra-prediction circuit 22 in a case where the image to bedecoded is an intra-coded image, and outputs a motion vector and anidentification flag to the motion prediction/compensation circuit 21 ina case where the image to be decoded is an inter-coded image.

In step S2, the dequantization circuit 13 performs dequantization usinga scheme corresponding to the quantization scheme used on the encodingside, and outputs the transform coefficient to the inverse orthogonaltransform circuit 14.

In step S3, the inverse orthogonal transform circuit 14 performs theinverse orthogonal transform on the transform coefficient supplied fromthe dequantization circuit 13, and outputs an obtained image to theadder circuit 15.

In step S4, the adder circuit 15 combines the decoded image suppliedfrom the inverse orthogonal transform circuit 14 and a prediction imagesupplied from the motion prediction/compensation circuit 21 or theintra-prediction circuit 22, and outputs a composite image to thedeblocking filter 16.

In step S5, the deblocking filter 16 performs filtering to remove blocknoise included in the composite image, and outputs an image from whichblock noise has been removed.

In step S6, the frame memory 19 temporarily stores the image suppliedfrom the deblocking filter 16.

In step S7, the control circuit 31 judges whether or not the targetimage is an intra-coded image.

In a case where it is judged in step S7 that the target image is anintra-coded image, then, in step S8, the intra-prediction circuit 22generates a prediction image by performing intra-prediction, and outputsthe generated prediction image to the adder circuit 15.

In contrast, in a case where it is judged in step S7 that the targetimage is not an intra-coded image, or is an inter-coded image, then, instep S9, the motion prediction/compensation circuit 21 performs a motionprediction/compensation process. A prediction image generated byperforming a motion prediction/compensation process is output to theadder circuit 15. The motion prediction/compensation process will bedescribed below with reference to a flowchart in FIG. 14.

In step S10, the control circuit 31 judges whether or not the aboveprocess has been performed on the macroblocks in one entire frame. In acase where it is judged that the process has not been performed,attention is focused on another macroblock and the process from step S1is repeatedly performed.

In contrast, in a case where it is judged in step S10 that the processhas been performed on the macroblocks in one entire frame, then, in stepS11, the rearrangement buffer 17 outputs a generated frame to the D/Aconverter circuit 18 in accordance with the control by the controlcircuit 31.

In step S12, the D/A converter circuit 18 performs D/A conversion on theframe supplied from the rearrangement buffer 17, and outputs an analogsignal to outside. The above process is performed on the individualframes.

Next, the motion prediction/compensation process performed in step S9 inFIG. 13 will be explained with reference to a flowchart in FIG. 14.

In step S31, the prediction mode determination circuit 41 of the motionprediction/compensation circuit 21 judges whether or not theidentification flag supplied from the lossless decoding circuit 12indicates that the process is to be performed in the filteringprediction mode.

In a case where it is judged in step S31 that the identification flagindicates that the process is to be performed in the filteringprediction mode, the process proceeds to step S32. In step S32, theprediction circuit 44 performs an extraction process for extracting amotion compensation image and a generation process for generating anintra-prediction image. The details of the extraction process and thegeneration process will be described below.

If a motion compensation image is extracted and an intra-predictionimage is generated, in step S33, the filtering circuit 45 performs afiltering prediction process. The details of the filtering predictionprocess will be described below.

When the processing of step S33 is completed, the motionprediction/compensation process ends. Then, the process returns to stepS9 in FIG. 11, and the process proceeds to step S10.

Furthermore, in a case where it is judged in step S31 that theidentification flag indicates does not indicate that the process is tobe performed in the filtering prediction mode, the process proceeds tostep S32. In step S32, the unidirectional prediction circuit 42 performsunidirectional prediction or the bidirectional prediction circuit 43performs bidirectional prediction. Thus, a prediction image isgenerated.

That is, in a case where the identification flag indicates that theprocess is to be performed in the unidirectional prediction mode, motionvectors are supplied from the prediction mode determination circuit 41to the unidirectional prediction circuit 42, and the unidirectionalprediction circuit 42 performs unidirectional prediction. Furthermore,in a case where the identification flag indicates that the process is tobe performed in the bidirectional prediction mode, motion vectors aresupplied from the prediction mode determination circuit 41 to thebidirectional prediction circuit 43, and the bidirectional predictioncircuit 43 performs bidirectional prediction. After the prediction imageis output to the adder circuit 15, the motion prediction/compensationprocess ends. Then, the process returns to step S9 in FIG. 13, and theprocess proceeds to step S10.

Next, an example of a flow of the extraction process executed in stepS32 in FIG. 14 will be explained with reference to a flowchart in FIG.15.

When the extraction process is started, in step S51, the predictioncircuit 44 sets a variable i=1. In step S52, the motion compensationcircuit 51 performs motion compensation from the i-th reference frame,that is, the reference plane i, and extracts a motion compensation imageMC[i]. In step S53, the motion compensation circuit 51 outputs themotion compensation image MC[i] to the filtering circuit 45.

In step S54, the prediction circuit 44 judges whether or not the valueof the variable i is less than or equal to N. In a case where it isjudged that the value of the variable i is less than or equal to thecertain natural number N, the process proceeds to step S55.

In step S55, the motion compensation circuit 51 increments the variablei. When the processing of step S55 is completed, the process returns tostep S52 and the subsequent process is repeated. Furthermore, in a casewhere it is judged in step S54 that the value of the variable i isgreater than N, the process proceeds to step S56.

In step S56, the intra-prediction circuit 52 judges whether or notintra-prediction is to be performed. In a case where it is judged thatintra-prediction is to be performed, the process proceeds to step S57.In step S57, the intra-prediction circuit 52 performs intra-predictionon the current frame (present frame). In step S58, the intra-predictioncircuit 52 outputs the intra-prediction image IP generated throughintra-prediction to the filtering circuit 45.

When the processing of step S58 is completed, the extraction processends. Then, the process returns to step S32 in FIG. 14, and the processproceeds to step S33. Furthermore, in a case where it is judged in stepS56 in FIG. 15 that intra-prediction is not to be performed, theextraction process ends. Then, the process returns to step S32 in FIG.14, and the process proceeds to step S33.

Next, an example of a flow of the filtering prediction process executedin step S33 in FIG. 14 will be explained with reference to a flowchartin FIG. 16.

After a motion compensation image is extracted and an intra-predictionimage is generated, when the filtering process is started, in step S71,the difference calculation circuit 61 of the filtering circuit 45calculates the difference between the motion compensation image and theintra-prediction image, and outputs a difference image to the low-passfilter circuit 62.

In step S72, the low-pass filter circuit 62 applies a low-pass filter tothe difference image supplied from the difference calculation circuit61, and outputs an obtained image to the gain adjustment circuit 63 andthe high-pass filter circuit 64.

In step S73, the gain adjustment circuit 63 adjusts the gain of theimage supplied from the low-pass filter circuit 62, and outputs thegain-adjusted image to the adder circuit 66.

In step S74, the high-pass filter circuit 64 applies a high-pass filterto the difference image supplied from the low-pass filter circuit 62,and outputs an obtained image to the gain adjustment circuit 65.

In step S75, the gain adjustment circuit 65 adjusts the gain of thedifference image supplied from the high-pass filter circuit 64, andoutputs the gain-adjusted image to the adder circuit 66.

In step S76, the adder circuit 66 adds together the image (output of thelow-pass filter) supplied from the gain adjustment circuit 63 and theimage (output of the high-pass filter) supplied from the gain adjustmentcircuit 65, and determines the high-frequency components of the images.The determined high-frequency components are supplied from the addercircuit 66 to the adder circuit 67.

In step S77, the adder circuit 67 adds the image (high-frequencycomponents) supplied from the adder circuit 66 to the intra-predictionimage, and outputs an obtained image to the adder circuit 15 as aprediction image. Note that the adder circuit 67 may add the image(high-frequency components) supplied from the adder circuit 66 to themotion compensation image in place of the intra-prediction image.

In step S78, the filtering circuit 45 judges whether or not all themotion compensation images and intra-prediction images have beenprocessed. In a case where it is judged that an unprocessed motioncompensation image or intra-prediction image exists, the process returnsto step S71 and the subsequent process is repeated.

Furthermore, in a case where it is judged in step S78 that all themotion compensation images and intra-prediction images have beenprocessed, the filtering prediction process ends. Then, the processreturns to step S33 in FIG. 12, and the motion prediction/compensationprocess ends. Then, the process returns to in step S9 in FIG. 11, andthe process proceeds to step S10.

As above, since decoding is performed using a prediction image generatedthrough filtering prediction, the lack of prediction accuracy ofintra-prediction can be compensated for by using the prediction accuracyof inter-prediction, thus making it possible to obtain a high-definitiondecoded image. Furthermore, at this time, at least one intra-predictionimage is used for the computation of a prediction image, thus allowing areduction in the number of motion vectors to be encoded and realizing areduction in processing cost (preventing an increase in load). That isto say, the decoding apparatus 1 can generate a high-accuracy predictionimage with a small amount of control information.

2. Second Embodiment Configuration of Encoding Apparatus

Next, the configuration and operation of an apparatus on the encodingside will be explained.

FIG. 17 is a block diagram illustrating an example configuration of anencoding apparatus 101. Compressed image information obtained throughencoding by the encoding apparatus 101 is input to the decodingapparatus 1 in FIG. 5.

An A/D converter circuit 111 performs A/D conversion on an input signal,and outputs an image to the rearrangement buffer 112.

A rearrangement buffer 112 rearranges frames in accordance with the GOP(Group of Pictures) structure of the compressed image information, andoutputs an image of a certain unit such as a macroblock. The imageoutput from the rearrangement buffer 112 is supplied to an adder circuit113, a mode determination circuit 123, a motion prediction/compensationcircuit 125, and an intra-prediction circuit 126.

The adder circuit 113 determines the difference between the imagesupplied from the rearrangement buffer 112 and a prediction imagegenerated by the motion prediction/compensation circuit 125 or theintra-prediction circuit 126 and supplied via a switch 127, and outputsthe residual to an orthogonal transform circuit 114. The closer theprediction image is to an original image and the smaller the number ofresiduals determined here is, the smaller the amount of code to beassigned to residuals is and therefore the higher the encodingefficiency can be said to be.

The orthogonal transform circuit 114 performs the orthogonal transform,such as the discrete cosine transform or Karhunen-Loeve transform, onthe residual supplied from the adder circuit 113, and outputs atransform coefficient obtained by performing the orthogonal transform toa quantization circuit 115.

The quantization circuit 115 quantizes the transform coefficientsupplied from the orthogonal transform circuit 114 in accordance withthe control by a rate control circuit 118, and outputs the quantizedtransform coefficient. The transform coefficient quantized by thequantization circuit 115 is supplied to the lossless encoding circuit116 and a dequantization circuit 119.

The lossless encoding circuit 116 compresses the transform coefficientsupplied from the quantization circuit 115 by performing losslessencoding such as variable-length encoding or arithmetic encoding, andoutputs information to a storage buffer 117.

Furthermore, the lossless encoding circuit 116 sets the value of anidentification flag in accordance with the information supplied from themode determination circuit 123, and describes the identification flag inthe header of the image. As described above, the decoding apparatus 1determines a prediction mode on the basis of the identification flagdescribed by the lossless encoding circuit 116.

The lossless encoding circuit 116 also describes the informationsupplied from the motion prediction/compensation circuit 125 or theintra-prediction circuit 126 in the header of the image. Motion vectorsand the like that are detected when inter-prediction is performed aresupplied from the motion prediction/compensation circuit 125, and theinformation regarding the intra-prediction mode applied is supplied fromthe intra-prediction circuit 126.

The storage buffer 117 temporarily stores the information supplied fromthe lossless encoding circuit 116, and outputs it as compressed imageinformation at a certain timing. The storage buffer 117 outputsinformation about the amount of code produced to the rate controlcircuit 118.

The rate control circuit 118 computes a quantization scale on the basisof the amount of code output from the storage buffer 117, and controlsthe quantization circuit 115 so that the quantization can be performedwith the computed quantization scale.

The dequantization circuit 119 performs dequantization on the transformcoefficient quantized by the quantization circuit 115, and outputs thetransform coefficient to an inverse orthogonal transform circuit 120.

The inverse orthogonal transform circuit 120 performs the inverseorthogonal transform on the transform coefficient supplied from thedequantization circuit 119, and outputs an obtained image to adeblocking filter 121.

The deblocking filter 121 removes block noise that appears in a locallydecoded image, and outputs an image from which the block noise has beenremoved to a frame memory 122.

The frame memory 122 stores the image supplied from the deblockingfilter 121. The image stored in the frame memory 122 is read by a modedetermination circuit 123 as appropriate.

The mode determination circuit 123 determines whether intra-coding is tobe performed or inter-coding is to be performed, on the basis of theimage stored in the frame memory 122 and the original image suppliedfrom the rearrangement buffer 112. Furthermore, in a case where it isdetermined that inter-coding is to be performed, the mode determinationcircuit 123 determines one mode among the unidirectional predictionmode, the bidirectional prediction mode, and the filtering predictionmode. The mode determination circuit 123 outputs information indicatingthe determination result to the lossless encoding circuit 116 as modeinformation.

In a case where it is determined that inter-coding is to be performed,the mode determination circuit 123 outputs frames that are stored in theframe memory 122 and that are obtained through local decoding to themotion prediction/compensation circuit 125 via a switch 124.

Furthermore, in a case where it is determined that intra-coding is to beperformed, the mode determination circuit 123 outputs frames that arestored in the frame memory 122 and that are obtained through localdecoding to the intra-prediction circuit 126.

The switch 124 is connected to a terminal a11 in a case whereinter-coding is to be performed, and is connected to a terminal b11 in acase where intra-coding is to be performed. The switching of the switch124 is controlled by, for example, a control circuit 131.

The motion prediction/compensation circuit 125 detects motion vectors onthe basis of the original image supplied from the rearrangement buffer112 and reference frames read from the frame memory 122, and outputs thedetected motion vectors to the lossless encoding circuit 116.Furthermore, the motion prediction/compensation circuit 125 generates aprediction image by performing motion compensation using the detectedmotion vectors and the reference frames, and outputs the generatedprediction image to the adder circuit 113 via the switch 127.

The intra-prediction circuit 126 performs intra-prediction on the basisof the original image supplied from the rearrangement buffer 112 and areference frame locally decoded and stored in the frame memory 122, andgenerates a prediction image. The intra-prediction circuit 126 outputsthe generated prediction image to the adder circuit 113 via the switch127, and outputs intra-prediction mode information to the losslessencoding circuit 116.

The switch 127 is connected to a terminal a12 or a terminal b12, andoutputs a prediction image generated by the motionprediction/compensation circuit 125 or the intra-prediction circuit 126to the adder circuit 113.

The control circuit 131 controls the overall operation of the encodingapparatus 101 by, for example, switching the connection of the switches124 and 127 in accordance with the mode determined by the modedetermination circuit 123.

FIG. 18 is a block diagram illustrating an example configuration of themode determination circuit 123 in FIG. 17.

As illustrated in FIG. 18, the mode determination circuit 123 iscomposed of an intra-prediction circuit 141, an inter-prediction circuit142, a prediction error calculation circuit 143, and a determinationcircuit 144. In the mode determination circuit 123, each ofintra-prediction and inter-prediction is performed on a block having adifferent size, and which prediction mode prediction is to be performedin is determined from the obtained results. As for inter-prediction, theprocess is performed in each of the prediction modes, i.e., theunidirectional prediction mode, the bidirectional prediction mode, andthe filtering prediction mode. The original image supplied from therearrangement buffer 112 is input to the intra-prediction circuit 141,the inter-prediction circuit 142, and the prediction error calculationcircuit 143.

The intra-prediction circuit 141 performs intra-prediction in units ofblocks having different sizes on the basis of the original image and animage read from the frame memory 122, and outputs a generatedintra-prediction image to the prediction error calculation circuit 143.A 4×4 prediction circuit 151-1 performs intra-prediction in units ofblocks of 4×4 pixels, and an 8×8 prediction circuit 151-2 performsintra-prediction in units of blocks of 8×8 pixels. A 16×16 predictioncircuit 151-3 performs intra-prediction in units of blocks of 16×16pixels. Each circuit of the intra-prediction circuit 141 also suppliesthe generated intra-prediction image to the filtering circuit 164.

A prediction circuit 161 in the inter-prediction circuit 142 detectsmotion vectors in units of blocks each having a different size on thebasis of the original image and reference frames read from the framememory 122. Furthermore, the prediction circuit 161 performs motioncompensation on the basis of the detected motion vectors, and outputsmotion compensation images that are used for the generation of aprediction image.

A 16×16 prediction circuit 161-1 performs a process on an image of everyblock of 16×16 pixels, and a 16×8 prediction circuit 161-2 performs aprocess on an image of every block of 16×8 pixels. Furthermore, a 4×4prediction circuit 161-(n−1) performs a process on an image of everyblock of 4×4 pixels. A skip/direct prediction circuit 161-n detectsmotion vectors in a skip prediction mode or a direct prediction mode,and performs motion compensation using the detected motion vectors.

Motion compensation images extracted from a plurality of referenceframes located in either direction with respect to the present frame aresupplied from the respective circuits in the prediction circuit 161 to aunidirectional prediction circuit 162. Furthermore, motion compensationimages extracted from a plurality of reference frames located in bothdirections with respect to the present frame are supplied from therespective circuits in the prediction circuit 161 to a bidirectionalprediction circuit 163.

In a case where filtering prediction is to be performed using, asdescribed above, motion compensation images extracted from a pluralityof reference frames located in either direction, motion compensationimages that are extracted from reference frames located in eitherdirection are supplied from the respective circuits in the predictioncircuit 161 to a filtering circuit 164. In a case where filteringprediction is to be performed using motion compensation images extractedfrom a plurality of reference frames located in both directions, motioncompensation images extracted from reference frames located in bothdirections are supplied from the respective circuits in the predictioncircuit 161 to the filtering circuit 164.

The unidirectional prediction circuit 162 generates a prediction imageby performing unidirectional prediction using the motion compensationimages having different sizes, which are supplied from the respectivecircuits in the prediction circuit 161, and outputs the generatedprediction image to the prediction error calculation circuit 143. Forexample, the unidirectional prediction circuit 162 generates aprediction image by using a pixel value in one image among a pluralityof motion compensation images having 16×16 pixels, which are suppliedfrom the prediction circuit 161-1, as a pixel value in the predictionimage.

The bidirectional prediction circuit 163 generates a prediction image byperforming bidirectional prediction using the motion compensation imageshaving different sizes, which are supplied from the respective circuitsin the prediction circuit 161, and outputs the generated predictionimage to the prediction error calculation circuit 143. For example, thebidirectional prediction circuit 163 generates a prediction image byusing the average value of pixel values in a plurality of motioncompensation images having 16×16 pixels, which are supplied from theprediction circuit 161-1, as a pixel value in the prediction image.

The filtering circuit 164 generates a prediction image by performingfiltering prediction using motion compensation images having differentsize, which are supplied from the respective circuits in the predictioncircuit 161, and intra-prediction images having different sizes, whichare supplied from the respective circuits in the intra-predictioncircuit 141, and outputs the generated prediction image to theprediction error calculation circuit 143. The filtering circuit 164 hasthe same configuration as the configuration illustrated in FIG. 12 so asto correspond to the filtering circuit 45 of the decoding apparatus 1.

For example, in the case of generating a prediction image based on amotion compensation image of 16×16 pixels, which is supplied from theprediction circuit 161-1, and based on 16×16 pixels, which are suppliedfrom the prediction circuit 151-3, the filtering circuit 164 determinesthe difference between the intra-prediction image and the motioncompensation image, and applies a low-pass filter to a determineddifference image. Furthermore, the filtering circuit 164 applies ahigh-pass filter to the output of the low-pass filter, and adds togetheran image obtained by adjusting the gain of the output of the high-passfilter and an image obtained by adjusting the gain of the output of thelow-pass filter. The filtering circuit 164 generates a prediction imageby adding the image of the addition result representing thehigh-frequency components to the intra-prediction image, and outputs thegenerated prediction image to the prediction error calculation circuit143.

The prediction error calculation circuit 143 determines the differencebetween each of the prediction images supplied from the respectivecircuits in the intra-prediction circuit 141 and the original image, andoutputs residual signals representing the determined difference to thedetermination circuit 144. Furthermore, the prediction error calculationcircuit 143 determines the difference between each of the predictionimages supplied from the unidirectional prediction circuit 162, thebidirectional prediction circuit 163, and the filtering circuit 164 inthe inter-prediction circuit 142 and the original image, and outputsresidual signals representing the determined difference to thedetermination circuit 144.

The determination circuit 144 measures the intensities of the residualsignals supplied from the prediction error calculation circuit 143, anddetermines, as a prediction method for generating a prediction image tobe used for encoding, a prediction method that has been used to generatea prediction image having a small difference from the original image.The determination circuit 144 outputs information representing thedetermination result to the lossless encoding circuit 116 as modeinformation. The mode information also includes information representingthe size of a block to be used as a unit of processing, and the like.

Furthermore, in a case where it is determined that a prediction image isto be generated through inter-prediction (in a case where it isdetermined that inter-coding is to be performed), the determinationcircuit 144 outputs reference frames read from the frame memory 122,together with the mode information, to the motionprediction/compensation circuit 125. In a case where it is determinedthat a prediction image is to be generated through intra-prediction (ina case where it is determined that intra-coding is to be performed), thedetermination circuit 144 outputs an image used for intra-prediction,which is read from the frame memory 122, to the intra-prediction circuit126 together with the mode information.

FIG. 19 is a block diagram illustrating an example configuration of themotion prediction/compensation circuit 125 in FIG. 17.

As illustrated in FIG. 19, the motion prediction/compensation circuit125 is composed of a motion vector detection circuit 181, aunidirectional prediction circuit 182, a bidirectional predictioncircuit 183, a prediction circuit 184, and a filtering circuit 185. Themotion prediction/compensation circuit 125 has a configuration similarto that of the motion prediction/compensation circuit 21 illustrated inFIG. 8, except that the motion vector detection circuit 181 is providedin place of the prediction mode determination circuit 41.

The motion vector detection circuit 181 detects motion vectors byperforming block matching or the like on the basis of the original imagesupplied from the rearrangement buffer 112 and reference frames suppliedfrom the mode determination circuit 123. The motion vector detectioncircuit 181 refers to the mode information supplied from the modedetermination circuit 123, and outputs the motion vectors, together withthe reference frames, to one of the unidirectional prediction circuit182, the bidirectional prediction circuit 183, and the predictioncircuit 184.

The motion vector detection circuit 181 outputs the motion vectors,together with the reference frames, to the unidirectional predictioncircuit 182 in a case where it has been selected that unidirectionalprediction is to be performed, and outputs these pieces of informationto the bidirectional prediction circuit 183 in a case where it has beenselected that bidirectional prediction is to be performed. In a casewhere it has been selected that filtering prediction is to be performed,the motion vector detection circuit 181 outputs the motion vectors,together with the reference frames, to the prediction circuit 184.

The unidirectional prediction circuit 182 generates a prediction imageby, similarly to the unidirectional prediction circuit 42 in FIG. 8,performing unidirectional prediction. The unidirectional predictioncircuit 182 outputs the generated prediction image to the adder circuit113.

The bidirectional prediction circuit 183 generates a prediction imageby, similarly to the bidirectional prediction circuit 43 in FIG. 8,performing bidirectional prediction. The bidirectional predictioncircuit 183 outputs the generated prediction image to the adder circuit113.

Similarly to the prediction circuit 44 in FIG. 8, the prediction circuit184 extracts a motion compensation image from each of a plurality ofreference frames such as two reference frames, and outputs the pluralityof extracted motion compensation images to the filtering circuit 185.

Similarly to the filtering circuit 45 in FIG. 8, the filtering circuit185 generates a prediction image by performing filtering prediction. Thefiltering circuit 185 outputs the generated prediction image to theadder circuit 113. Note that the filtering circuit 185 has aconfiguration similar to the configuration of the filtering circuit 45illustrated in FIG. 12. Hereinafter, the configuration of the filteringcircuit 185 will be explained by referring to the configuration of thefiltering circuit 45 illustrated in FIG. 12, as appropriate.

A prediction image generated through filtering prediction may be animage that includes more high-frequency components than a predictionimage generated through unidirectional prediction or bidirectionalprediction and that has a small difference from the original image.Therefore, only a small amount of code to be assigned to residuals isrequired, thus making it possible to increase encoding efficiency.

Furthermore, since filtering prediction can be performed using at leasttwo reference frames, such an increase in encoding efficiency is madefeasible without increasing the complexity of processing. The residualsfrom the original image can be reduced and the encoding efficiency canbe increased also by, for example, increasing the number of referenceframes to be used for inter-prediction to generate a high-accuracyprediction image and by using it. However, in this case, the number ofreference frame used increases, and the complexity of processingincreases.

Note that when a prediction method is selected, an optimum predictionmethod may be selected by taking into account the amount of code forinformation such as motion vectors necessary for prediction and anencoding mode and by adding a weight corresponding to the amount of codeto the intensity of a residual signal. This makes it possible to furtherimprove encoding efficiency. Furthermore, for a simplified encodingprocess, a prediction method may be adaptively selected utilizingfeature values of an input original image in the time and spacedirections.

[Explanation of Flow of Encoding Process]

Next, a process of the encoding apparatus 101 having the aboveconfiguration will be explained.

An encoding process of the encoding apparatus 101 will be explained withreference to a flowchart in FIG. 20. This process is started when animage of a certain unit such as a macroblock is output from therearrangement buffer 112.

In step S101, the adder circuit 113 determines the difference between animage supplied from the rearrangement buffer 112 and a prediction imagegenerated by the motion prediction/compensation circuit 125 or theintra-prediction circuit 126, and outputs the residual to the orthogonaltransform circuit 114.

In step S102, the orthogonal transform circuit 114 performs theorthogonal transform on the residual supplied from the adder circuit113, and outputs a transform coefficient to the quantization circuits115.

In step S103, the quantization circuit 115 quantizes the transformcoefficient supplied from the orthogonal transform circuit 114, andoutputs the quantized transform coefficient.

In step S104, the dequantization circuit 119 performs dequantization onthe transform coefficient quantized by the quantization circuit 115, andoutputs the transform coefficient to the inverse orthogonal transformcircuit 120.

In step S105, the inverse orthogonal transform circuit 120 performs theinverse orthogonal transform on the transform coefficient supplied fromthe dequantization circuit 119, and outputs an obtained image to thedeblocking filter 121.

In step S106, the deblocking filter 121 removes block noise byperforming filtering, and outputs an image from which the block noisehas been removed to the frame memory 122.

In step S107, the frame memory 122 stores the image supplied from thedeblocking filter 121.

In step S108, the mode determination circuit 123 performs a modedetermination process. With the mode determination process, whichprediction mode a prediction image is to be generated in is determined.The mode determination process will be described below.

In step S109, the control circuit 131 judges whether or notintra-prediction is to be performed on the basis of the determination bythe mode determination circuit 123.

In a case where it is judged in step S109 that intra-prediction is to beperformed, then, in step S110, the intra-prediction circuit 126 performsintra-prediction, and outputs a prediction image to the adder circuit113.

In contrast, in a case where it is judged in step S109 thatintra-prediction is not to be performed, that is, inter-prediction is tobe performed, then, in step Sill, a motion prediction/compensationprocess is performed by the motion prediction/compensation circuit 125,and a prediction image is output to the adder circuit 113. The motionprediction/compensation process will be described below.

In step S112, the lossless encoding circuit 116 compresses the transformcoefficient supplied from the quantization circuit 115, and outputs itto the storage buffer 117. Furthermore, the lossless encoding circuit116 describes an identification flag in the header of the image ordescribes motion vectors supplied from the motionprediction/compensation circuit 125 in the header of the image inaccordance with the information supplied from the mode determinationcircuit 123.

In step S113, the storage buffer 117 temporarily stores the informationsupplied from the lossless encoding circuit 116.

In step S114, the control circuit 131 judges whether or not the aboveprocess has been performed on the macroblocks in one entire frame. In acase where it is judged that the process has not been performed,attention is focused on another macroblock and the process from stepS111 is repeatedly performed.

In contrast, in a case where it is judged in step S114 that the processhas been performed on the macroblocks in one entire frame, then, in stepS115, the storage buffer 117 outputs compressed image information inaccordance with the control by the control circuit 131. The aboveprocess is performed on each frame.

Next, the mode determination process performed in step S108 in FIG. 20will be explained with reference to a flowchart in FIG. 21.

In step S131, the intra-prediction circuit 141 and the inter-predictioncircuit 142 perform intra-prediction and inter-prediction, respectively,on blocks having different dimensions, and generate prediction images.The generated prediction images are supplied to the prediction errorcalculation circuit 143.

In step S132, the prediction error calculation circuit 143 determinesthe difference between each of the prediction images supplied from therespective circuits in the intra-prediction circuit 141, and theunidirectional prediction circuit 162, bidirectional prediction circuit163, and filtering circuit 164 of the inter-prediction circuit 142 andthe original image. The prediction error calculation circuit 143 outputsresidual signals to the determination circuit 144.

In step S133, the determination circuit 144 determines a predictionmethod for generating a prediction image to be supplied to the addercircuit 113 on the basis of the intensities of the residual signalssupplied from the prediction error calculation circuit 143.

In step S134, the determination circuit 144 outputs mode informationthat is information regarding the determined prediction method to thelossless encoding circuit 116. After that, the process returns to stepS108 in FIG. 20, and the subsequent processing is performed.

Next, the motion prediction/compensation process performed in step S111in FIG. 20 will be described with reference to a flowchart in FIG. 22.

In step S151, the motion vector detection circuit 181 detects motionvectors on the basis of the original image and a reference frame.

In step S152, the motion vector detection circuit 181 judges whether ornot the mode determination circuit 123 has determined that the processis to be performed in the filtering prediction mode.

In a case where it is judged that it has been determined that theprocess is to be performed in the filtering prediction mode, the processproceeds to step S153. The processing of steps S153 and S154 is executedin a manner similar to the case in steps S32 and S33 in FIG. 14,respectively. That is to say, in step S153, the extraction process isexecuted in a manner as explained with reference to the flowchart inFIG. 15, and in step S154, the filtering prediction process is executedin a manner as explained with reference to the flowchart in FIG. 16.

When the processing of step S154 is completed, the motionprediction/compensation process ends. Then, the process returns to stepSill in FIG. 20, and the process proceeds to step S112.

Furthermore, in a case where it is judged in step S152 in FIG. 22 thatit has not been determined that the process is to be performed in thefiltering prediction mode, the process proceeds to step S155. In stepS155, the unidirectional prediction circuit 182 or the bidirectionalprediction circuit 183 performs unidirectional prediction orbidirectional prediction, and generates a prediction image.

That is, in a case where it has been determined that the process is tobe performed in the unidirectional prediction mode, motion vectors aresupplied from the motion vector detection circuit 181 to theunidirectional prediction circuit 182, and unidirectional prediction isperformed by the unidirectional prediction circuit 182. Furthermore, ina case where it has been determined that the process is to be performedin the bidirectional prediction mode, motion vectors are supplied fromthe motion vector detection circuit 181 to the bidirectional predictioncircuit 183, and bidirectional prediction is performed by thebidirectional prediction circuit 183. When the prediction image isoutput to the adder circuit 113 and the processing of step S155 in FIG.22 is completed, the motion prediction/compensation process ends. Then,the process returns to step S111 in FIG. 20, and the process proceeds tostep S112.

As above, encoding is performed using a prediction image generatedthrough filtering prediction, thus making it possible to increaseencoding efficiency. In particular, the encoding apparatus 101 canimprove encoding efficiency by compensating for allowing the lack ofprediction accuracy of intra-prediction by using the prediction accuracyof inter-prediction. Moreover, the number of reference frames necessaryfor inter-prediction can be reduced, and therefore the reduction inprocessing cost can be realized.

3. Third Embodiment Modifications of Filtering Circuit

In the foregoing description, it is assumed that the filtering circuits45 and 185 are designed to have the configuration as illustrated in FIG.12. However, this configuration can be changed as appropriate.

FIG. 23 is a block diagram illustrating another example configuration ofthe filtering circuit 45. The configuration elements corresponding tothe configuration elements illustrated in FIG. 12 are assigned the samenumerals. Redundant explanation will be omitted as appropriate.

The difference calculation circuit 61 in FIG. 23 calculates thedifference between the intra-prediction image and the motioncompensation image, and outputs a difference image to the low-passfilter circuit 62.

The low-pass filter circuit 62 applies a low-pass filter to thedifference image supplied from the difference calculation circuit 61,and outputs an obtained image to the adder circuit 67.

The adder circuit 67 adds the image supplied from the low-pass filtercircuit 62 to the intra-prediction image, and outputs an obtained imageas a prediction image.

With the use of the configuration as illustrated in FIG. 23, the amountof processing can be reduced compared to that in the case where theconfiguration in FIG. 12 is used, and the high-speed operation is madefeasible.

FIG. 24 is a block diagram illustrating still another exampleconfiguration of the filtering circuit 45. The configuration elementscorresponding to the configuration elements illustrated in FIG. 12 areassigned the same numerals. Redundant explanation will be omitted asappropriate.

In the filtering circuit 45 in FIG. 24, filtering is performed not ontime domain signals but on frequency domain signals. Both the filteringcircuits 45 illustrated in FIGS. 12 and 23 are configured to performfiltering on time domain signals.

The difference calculation circuit 61 in FIG. 24 calculates thedifference between the intra-prediction image and the motioncompensation image, and outputs a difference image to an orthogonaltransform circuit 201.

The orthogonal transform circuit 201 performs the orthogonal transform,represented by the DCT (Discrete Cosine Transform), Hadamard transform,and KLT (Karhunen Loeve Transformation), on the difference image, andoutputs a signal obtained after the orthogonal transform to a band-passfilter circuit 202. Performing the orthogonal transform and performingfiltering on signals in the frequency domain make it possible to moreflexibly implement a high-accuracy filtering process than that in a casewhere filtering is performed on signals in the time domain.

In a case where the DCT is used as the orthogonal transform, an outputDF obtained after the orthogonal transform is represented by Equation(10) below. In Equation (10), DCT(X) represents the performance of atwo-dimensional DCT process on a signal X.

[Math. 10]

DF=DCT(D)  (10)

The band-pass filter circuit 202 performs filtering on the output of theorthogonal transform circuit 201, and outputs a signal in a certainband.

A gain adjustment circuit 203 adjusts the gain of the output of theband-pass filter circuit 202 by multiplying it by a, and also performsfrequency component adjustment. An output XF of the gain adjustmentcircuit 203 is represented by Equation (11) below. In Equation (11),BPF(X) represents the performance of a band-pass filtering process on asignal X.

[Math. 11]

XF=α·BPF(DF)  (11)

An inverse orthogonal transform circuit 204 performs the inverseorthogonal transform using a scheme corresponding to the orthogonaltransform performed by the orthogonal transform circuit 201, andtransforms the frequency domain signal supplied from the gain adjustmentcircuit 203 into a time domain signal. For example, in a case where theorthogonal transform circuit 201 uses the DCT as the orthogonaltransform, the inverse orthogonal transform circuit 204 performs theIDCT. An output X of the inverse orthogonal transform circuit 204 isrepresented by Equation (12) below. In Equation (12), IDCT(X) representsthe performance of a two-dimensional IDCT process on the signal X.

[Math. 12]

X=IDCT(XF)  (12)

The adder circuit 67 adds the signal X supplied from the inverseorthogonal transform circuit 204 to the intra-prediction image in thetime domain, and outputs an obtained image as a prediction image. Aprediction image S(i, j) that is the final output of the adder circuit67 is represented by Equation (13) as follows.

[Math. 13]

S(i,j)=MC₀(i,j)+X(i,j)  (13)

In this manner, even if filtering is performed on a frequency domainsignal, a high-accuracy prediction image can be generated.

Furthermore, in the foregoing description, it is assumed that filteringprediction is performed using two reference frames. However, a largernumber of frames may be used as reference frames.

FIG. 25 is a diagram illustrating an example in a case where threereference frames are used.

In the example in FIG. 25, the temporally preceding frame with respectto the time of a prediction frame, the further preceding frame, and thestill further preceding frame, i.e., three frames, are set as referenceframes. The preceding frame closer to the prediction frame is set as areference frame R0, the preceding frame with respect to the referenceframe R0 is set as a reference frame R1, and the preceding frame withrespect to the reference frame R1 is set as a reference frame R2.

FIG. 26 is a block diagram illustrating an example configuration of afiltering circuit in a case where three reference frames are used.

As illustrated in FIG. 26, a filtering circuit 211 is composed of afiltering circuit 221 and a filtering circuit 222. Each of the filteringcircuit 221 and the filtering circuit 222 has the configuration asillustrated in FIG. 12, 23, or 24. That is, the filtering circuit 211 isconfigured to operate as a three-input and one-output circuit byconnecting, in cascade, filtering circuits 45 used for the two-input andone-output design.

Here, the explanation will be given assuming that a motion compensationimage extracted from the reference frame R0 is set as the motioncompensation image MC0, a motion compensation image extracted from thereference frame R1 is set as the motion compensation image MC1, and anintra-prediction image is obtained in the current frame. Theintra-prediction image IP and the motion compensation image MC0 areinput to the filtering circuit 221 and the motion compensation image MC1is input to the filtering circuit 222.

The filtering circuit 221 performs filtering on the intra-predictionimage IP and the motion compensation image MC0 in a manner as explainedwith reference to FIG. 12, and outputs an intermediate output X that isa result of filtering to the filtering circuit 222.

The filtering circuit 221 performs filtering on the intermediate outputX and the motion compensation image MC1 in a manner as explained withreference to FIG. 12, and outputs a result of filtering as a predictionimage.

The filtering circuit 211 that handles such three frames, in place ofthe filtering circuit 45, can also be provided in the decoding apparatus1 in FIG. 5 or the encoding apparatus 101 in FIG. 17.

Note that the filtering circuit 221 and the filtering circuit 222 maynot necessarily have the same configuration, and may have differentconfigurations such that one of them has the configuration illustratedin FIG. 12 and the other has the configuration illustrated in FIG. 23.Furthermore, parameters used for the filters can also be made differentfrom each other while taking into account the input/outputcharacteristics obtained before and after filtering.

The filtering circuit 211 may perform filtering not on motioncompensation images extracted from reference frames located in onetemporal direction but on motion compensation images extracted fromthree reference frames located in the forward and backward directions.

Note that in a case where the preceding and following frames withrespect to the time of a prediction frame are used as reference frames,including the case explained with reference to FIG. 12, a parameter suchas a tap coefficient during filtering may be dynamically changed inaccordance with the time directions of the reference frames or thedistances therebetween.

Compressed image information is transmitted from the encoding apparatus101 to the decoding apparatus 1 via various media including recordingmedia such as an optical disk, a magnetic disk, and a flash memory,satellite broadcasting, cable TV, the Internet, and a mobile phonenetwork.

The series of processes described above can be executed by hardware orsoftware. In a case where the series of processes is executed software,a program constituting the software is installed from a programrecording medium into a computer incorporated in dedicated hardware, ora general-purpose personal computer or the like that is capable ofexecuting various functions by installing various programs therein.

FIG. 27 is a block diagram illustrating an example configuration ofhardware of a computer 300 that executes the series of processesdescribed above in accordance with a program.

A CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, and aRAM (Random Access Memory) 303 are interconnected via a bus 304.

An input/output interface 310 is further connected to the bus 304. Aninput unit 311 including a keyboard, a mouse, a microphone, etc., anoutput unit 312 including a display, a speaker, etc., a storage unit 313formed of a hard disk, a non-volatile memory, etc., a communication unit314 formed of a network interface, etc., and a drive 315 that drives aremovable medium 321 such as an optical disk or a semiconductor memoryare connected to the input/output interface 310.

In the computer 300 configured as above, the CPU 301 loads a programstored in, for example, the storage unit 313 into the RAM 303 via theinput/output interface 310 and the bus 304 and executes the program,thereby performing the series of processes described above.

The program executed by the CPU 301 is provided by being recorded on,for example, the removable medium 321 or via a wired or wirelesstransmission medium such as a local area network, the Internet, ordigital broadcasting, and is installed into the storage unit 313.

Note that the program executed by the computer may be a program in whichprocesses are performed in a time-series manner in accordance with theorder explained herein, or may be a program in which processes areperformed in parallel or at a necessary timing such as when called.

Furthermore, as used herein, the steps describing a program recorded ona recording medium include, of course, processes performed in atime-series manner in the order described herein and also includeprocesses that are not necessarily processed in a time-series manner butare executed in parallel or individually.

Furthermore, as used herein, the term system refers to the overallapparatus composed of a plurality of devices (apparatuses).

Furthermore, the configuration explained above as a single apparatus (orprocessing unit) may be divided to construct a plurality of apparatuses(or processing units). Conversely, the configuration explained above asa plurality of apparatuses (or processing units) may be collected toconstruct a single apparatus (or processing unit). Furthermore, ofcourse, a configuration other than that described above may be added tothe configuration of each apparatus (or each processing unit). Moreover,a portion of the configuration of a certain apparatus (or processingunit) may be included in the configuration of another apparatus (oranother processing unit) if the configuration or operation as an entiresystem is substantially the same. That is to say, embodiments of thepresent invention are not to be limited to the embodiments describedabove, and a variety of changes can be made without deviating from thegist of the present invention.

For example, the decoding apparatus 1 or encoding apparatus 101described above can be applied to any electronic device. Examplesthereof will be explained hereinafter.

FIG. 28 is a block diagram illustrating an example configuration of amain part of a television receiver that uses the decoding apparatus 1 towhich the present invention is applied.

A television receiver 1000 illustrated in FIG. 28 includes a terrestrialtuner 1013, a video decoder 1015, a video signal processing circuit1018, a graphic generation circuit 1019, a panel drive circuit 1020, anda display panel 1021.

The terrestrial tuner 1013 receives a broadcast wave signal of aterrestrial analog broadcast via an antenna, demodulates it, obtains avideo signal, and supplies it to the video decoder 1015. The videodecoder 1015 performs a decoding process on the video signal suppliedfrom the terrestrial tuner 1013, and supplies an obtained digitalcomponent signal to the video signal processing circuit 1018.

The video signal processing circuit 1018 performs a certain process suchas noise removal on the video data supplied from the video decoder 1015,and supplies obtained video data to the graphic generation circuit 1019.

The graphic generation circuit 1019 generates video data of a program tobe displayed on the display panel 1021, image data obtained through aprocess that is based on an application supplied via a network, or thelike, and supplies the generated video data or image data to the paneldrive circuit 1020. Furthermore, the graphic generation circuit 1019also performs processes, as appropriate, such as generating video data(graphic) for displaying a screen utilized by a user to select an itemor the like, superimposing it onto the video data of the program toobtain video data, and supplying the obtained video data to the paneldrive circuit 1020.

The panel drive circuit 1020 drives the display panel 1021 on the basisof the data supplied from the graphic generation circuit 1019, andcauses video of a program or the various screens described above to bedisplayed on the display panel 1021.

The display panel 1021 is formed of an LCD (Liquid Crystal Display) orthe like, and causes video of a program or the like to be displayed inaccordance with the control by the panel drive circuit 1020.

Furthermore, the television receiver 1000 also includes an audio A/D(Analog/Digital) converter circuit 1014, an audio signal processingcircuit 1022, an echo cancellation/audio synthesis circuit 1023, anaudio amplifier circuit 1024, and a speaker 1025.

The terrestrial tuner 1013 demodulates a received broadcast wave signalto acquire a video signal as well as an audio signal. The terrestrialtuner 1013 supplies the acquired audio signal to the audio A/D convertercircuit 1014.

The audio A/D converter circuit 1014 performs an A/D conversion processon the audio signal supplied from the terrestrial tuner 1013, andsupplies an obtained digital audio signal to the audio signal processingcircuit 1022.

The audio signal processing circuit 1022 performs a certain process suchas noise removal on the audio data supplied from the audio A/D convertercircuit 1014, and supplies obtained audio data to the echocancellation/audio synthesis circuit 1023.

The echo cancellation/audio synthesis circuit 1023 supplies the audiodata supplied from the audio signal processing circuit 1022 to the audioamplifier circuit 1024.

The audio amplifier circuit 1024 performs a D/A conversion process andan amplification process on the audio data supplied from the echocancellation/audio synthesis circuit 1023, and adjusts it to a certainvolume before outputting audio from the speaker 1025.

Further, the television receiver 1000 also includes a digital tuner 1016and an MPEG decoder 1017.

The digital tuner 1016 receives a broadcast wave signal of a digitalbroadcast (terrestrial digital broadcast, BS (Broadcasting Satellite)/CS(Communications Satellite) digital broadcast) via the antenna,demodulates it, acquires an MPEG-TS (Moving Picture ExpertsGroup-Transport Stream), and supplies it to the MPEG decoder 1017.

The MPEG decoder 1017 descrambles the MPEG-TS supplied from the digitaltuner 1016, and extracts a stream including the data of a program to bereproduced (to be viewed and listened to). The MPEG decoder 1017 decodesaudio packets forming the extracted stream, and supplies obtained audiodata to the audio signal processing circuit 1022. Also, the MPEG decoder1017 decodes video packets forming the stream, and supplies obtainedvideo data to the video signal processing circuit 1018. Furthermore, theMPEG decoder 1017 supplies EPG (Electronic Program Guide) data extractedfrom the MPEG-TS to a CPU 1032 via a path that is not illustrated in thefigure.

The television receiver 1000 uses the decoding apparatus 1 describedabove as the MPEG decoder 1017 that decodes a video packet in the aboveway. Note that an MPEG-TS transmitted from a broadcast station or thelike has been encoded by the encoding apparatus 101.

Similarly to the case of the decoding apparatus 1, the MPEG decoder 1017obtains a motion prediction image from the frame (N−1) using a motionvector, and also obtains a spatial prediction image from the frame N byperforming intra-prediction by utilizing a decoded pixel value in theframe N. And the MPEG decoder 1017 performs a filtering process on thetwo types of images to generate a prediction image. Accordingly, theMPEG decoder 1017 can generate a new prediction image closer to theoriginal image.

The video data supplied from the MPEG decoder 1017 is subjected to,similarly to the case of the video data supplied from the video decoder1015, a certain process by the video signal processing circuit 1018, andgenerated video data or the like is superimposed thereon by the graphicgeneration circuit 1019. The resulting data is supplied to the displaypanel 1021 via the panel drive circuit 1020, and an image thereof isdisplayed.

The audio data supplied from the MPEG decoder 1017 is subjected to,similarly to the case of the audio data supplied from the audio A/Dconverter circuit 1014, a certain process by the audio signal processingcircuit 1022, supplied to the audio amplifier circuit 1024 via the echocancellation/audio synthesis circuit 1023, and subjected to a D/Aconversion process or an amplification process. Consequently, audiowhose volume has been adjusted to a certain value is output from thespeaker 1025.

Furthermore, the television receiver 1000 also includes a microphone1026 and an A/D converter circuit 1027.

The A/D converter circuit 1027 receives a signal of audio of a user,which is captured by the microphone 1026 provided in the televisionreceiver 1000 for use in audio conversation, performs an A/D conversionprocess on the received audio signal, and supplies obtained digitalaudio data to the echo cancellation/audio synthesis circuit 1023.

In a case where data of audio of a user (user A) of the televisionreceiver 1000 has been supplied from the A/D converter circuit 1027, theecho cancellation/audio synthesis circuit 1023 performs echocancellation on the audio data of the user A, and causes the data ofaudio obtained by, for example, being combined with other audio data tobe output from the speaker 1025 via the audio amplifier circuit 1024.

Further, the television receiver 1000 also includes an audio codec 1028,an internal bus 1029, an SDRAM (Synchronous Dynamic Random AccessMemory) 1030, a flash memory 1031, a CPU 1032, a USB (Universal SerialBus) I/F 1033, and a network I/F 1034.

The A/D converter circuit 1027 receives a signal of audio of a user,which is captured by the microphone 1026 provided in the televisionreceiver 1000 for use in audio conversation, performs an A/D conversionprocess on the received audio signal, and supplies obtained digitalaudio data to the audio codec 1028.

The audio codec 1028 converts the audio data supplied from the A/Dconverter circuit 1027 into data of a certain format for transmissionthrough a network, and supplies it to the network I/F 1034 via theinternal bus 1029.

The network I/F 1034 is connected to a network via a cable attached to anetwork terminal 1035. The network I/F 1034 transmits the audio datasupplied from the audio codec 1028 to, for example, another apparatusconnected to the network. Furthermore, the network I/F 1034 receives,for example, audio data transmitted from another apparatus connectedover a network via the network terminal 1035, and supplies it to theaudio codec 1028 via the internal bus 1029.

The audio codec 1028 converts the audio data supplied from the networkI/F 1034 into data of a certain format, and supplies it to the echocancellation/audio synthesis circuit 1023.

The echo cancellation/audio synthesis circuit 1023 performs echocancellation on the audio data supplied from the audio codec 1028, andcauses the data of audio obtained by, for example, being combined withother audio data to be output from the speaker 1025 via the audioamplifier circuit 1024.

The SDRAM 1030 stores various data necessary for the CPU 1032 to performprocesses.

The flash memory 1031 stores the program executed by the CPU 1032. Theprogram stored in the flash memory 1031 is read by the CPU 1032 at acertain timing such as when the television receiver 1000 is started. Theflash memory 1031 also stores EPG data acquired via digitalbroadcasting, data acquired from a certain server via a network, and thelike.

For example, the flash memory 1031 stores an MPEG-TS including contentdata acquired from a certain server over a network under the control ofthe CPU 1032. The flash memory 1031 supplies the MPEG-TS to the MPEGdecoder 1017 via the internal bus 1029 by, for example, the control ofthe CPU 1032.

The MPEG decoder 1017 processes the MPEG-TS in a manner similar to thatin the case of the MPEG-TS supplied from the digital tuner 1016. In thismanner, the television receiver 1000 can receive content data composedof video, audio, and the like over a network, decode the content datausing the MPEG decoder 1017, display video of the content data, andoutput audio.

Furthermore, the television receiver 1000 also includes a lightreceiving unit 1037 that receives light of an infrared signaltransmitted from a remote controller 1051.

The light receiving unit 1037 receives infrared light from the remotecontroller 1051, and outputs a control code indicating the content of auser operation obtained through demodulation to the CPU 1032.

The CPU 1032 executes a program stored in the flash memory 1031, andcontrols the overall operation of the television receiver 1000 inaccordance with the control code supplied from the light receiving unit1037 or the like. The CPU 1032 is connected to each unit of thetelevision receiver 1000 via a path that is not illustrated in thefigure.

The USB I/F 1033 transmits and receives data to and from an externaldevice of the television receiver 1000, which is connected via a USBcable attached to a USB terminal 1036. The network I/F 1034 is connectedto a network via a cable attached to the network terminal 1035, and alsotransmits and receives data other than audio data to and from variousapparatuses connected to the network.

With the use of the decoding apparatus 1 as the MPEG decoder 1017, thetelevision receiver 1000 can generate a prediction image by adding animage representing high-frequency components to the intra-predictionimage IP during the decoding on video packets forming a stream.

This prediction image includes more high-frequency components than aprediction image obtained in a case where bidirectional prediction issimply performed.

Furthermore, compared with the case where unidirectional prediction issimply performed, a prediction image can be generated by moreefficiently utilizing the temporal correlation of images.

Moreover, since a prediction image including a large number ofhigh-frequency components to a decoded image in the adder circuit 15,the image finally output from the MPEG decoder 1017 is also ahigh-definition image including a large number of high-frequencycomponents.

Furthermore, in the generation of a prediction image, intra-predictionis performed in the current frame, and a generated intra-predictionimage IP is utilized together with the motion compensation image MC1.Thus, the number of motion vectors for generating a motion compensationimage can be reduced.

In this manner, the television receiver 1000 can realize improvement inencoding efficiency by compensating for the lack of prediction accuracyof intra-prediction by using the prediction accuracy of inter-predictionduring the decoding on video packets forming a stream. Moreover, thereduction in processing cost can also be realized by reducing the numberof reference planes necessary for inter-prediction.

FIG. 29 is a block diagram illustrating an example configuration of amain part of a mobile phone that uses the decoding apparatus 1 and theencoding apparatus 101 to which the present invention is applied.

A mobile phone 1100 illustrated in FIG. 29 includes a main control unit1150 configured to totally control individual units, a power supplycircuit unit 1151, an operation input control unit 1152, an imageencoder 1153, a camera I/F unit 1154, an LCD control unit 1155, an imagedecoder 1156, a multiplexing/demultiplexing unit 1157, arecording/reproducing unit 1162, a modulation/demodulation circuit unit1158, and an audio codec 1159. They are interconnected via a bus 1160.

Furthermore, the mobile phone 1100 includes an operation key 1119, a CCD(Charge Coupled Devices) camera 1116, a liquid crystal display 1118, astorage unit 1123, a transmission/reception circuit unit 1163, anantenna 1114, a microphone (mic) 1121, and a speaker 1117.

When a call-end and power key is turned on by a user operation, thepower supply circuit unit 1151 supplies electric power to each unit froma battery pack, thereby starting the mobile phone 1100 so as to be ableto operate.

The mobile phone 1100 performs various operations in various modes suchas an audio call mode and a data communication mode, such astransmitting and receiving an audio signal, transmitting and receivingelectronic mail and image data, capturing images, and recording data, onthe basis of the control of the main control unit 1150 composed of aCPU, a ROM, a RAM, etc.

For example, in the audio call mode, the mobile phone 1100 converts,using the audio codec 1159, an audio signal collected by the microphone(mic) 1121 into digital audio data, performs a spread spectrum processon the digital audio data using the modulation/demodulation circuit unit1158, and performs a digital-to-analog conversion process and afrequency conversion process using the transmission/reception circuitunit 1163. The mobile phone 1100 transmits a transmission signalobtained through the conversion processes to a base station that is notillustrated in the figure via the antenna 1114. The transmission signal(audio signal) transmitted to the base station is supplied to a mobilephone on the other end of the call via a public telephone line network.

Furthermore, for example, in the audio call mode, the mobile phone 1100amplifies, using the transmission/reception circuit unit 1163, areceived signal that has been received by the antenna 1114, furtherperforms a frequency conversion process and an analog-to-digitalconversion process, performs an inverse spread spectrum process usingthe modulation/demodulation circuit unit 1158, and converts theresulting signal into an analog audio signal using the audio codec 1159.The mobile phone 1100 outputs the analog audio signal obtained byconversion from the speaker 1117.

Moreover, for example, in a case where electronic mail is transmitted inthe data communication mode, the mobile phone 1100 accepts, using theoperation input control unit 1152, text data of electronic mail input byoperating the operation key 1119. The mobile phone 1100 processes thetext data using the main control unit 1150, and causes the resultingdata to be displayed as an image on the liquid crystal display 1118 viathe LCD control unit 1155.

Furthermore, the mobile phone 1100 generates, using the main controlunit 1150, electronic mail data on the basis of the text data acceptedby the operation input control unit 1152, on the basis of a userinstruction, or the like. The mobile phone 1100 performs, using themodulation/demodulation circuit unit 1158, a spread spectrum process onthe electronic mail data, and performs, using the transmission/receptioncircuit unit 1163, a digital-to-analog conversion process and afrequency conversion process. The mobile phone 1100 transmits atransmission signal obtained by the conversion processes to a basestation that is not illustrated in the figure via the antenna 1114. Thetransmission signal (electronic mail) transmitted to the base station issupplied to a certain destination via a network, a mail server, and thelike.

Furthermore, for example, in a case where electronic mail is received inthe data communication mode, the mobile phone 1100 receives, using thetransmission/reception circuit unit 1163, a signal transmitted from abase station via the antenna 1114, amplifies it, and further performs afrequency conversion process and an analog-to-digital conversionprocess. The mobile phone 1100 performs an inverse spread spectrumprocess on the received signal using the modulation/demodulation circuitunit 1158 to restore the original electronic mail data. The mobile phone1100 displays the restored electronic mail data on the liquid crystaldisplay 1118 via the LCD control unit 1155.

Note that the mobile phone 1100 is also capable of recording (storing)received electronic mail data on the storage unit 1123 via therecording/reproducing unit 1162.

The storage unit 1123 is any rewritable storage medium. The storage unit1123 may be, for example, a semiconductor memory such as a RAM or abuilt-in flash memory, or may be a hard disk, or a removable medium suchas a magnetic disk, a magneto-optical disk, an optical disk, a USBmemory, or a memory card. Of course, any other type of medium may beused.

Further, for example, in a case where image data is transmitted in thedata communication mode, the mobile phone 1100 generates, using the CCDcamera 1116, image data by capturing an image. The CCD camera 1116includes optical devices such as a lens and an aperture, and a CCDserving as a photoelectric conversion element, captures an image of anobject, converts the intensity of received light into an electricalsignal, and generates image data of the image of the object. The CCDcamera 1116 encodes the image data using the image encoder 1153 via thecamera I/F unit 1154, and converts the image data into encoded imagedata.

The mobile phone 1100 uses the encoding apparatus 101 described above asthe image encoder 1153 that performs the above process. Similarly to thecase of the encoding apparatus 101, the image encoder 1153 obtains amotion prediction image from the frame (N−1) using motion vectors, and,in addition, obtains a spatial prediction image from the frame N throughintra-prediction utilizing a decoded pixel value in the frame N. And theimage encoder 1153 performs a filtering process on the two types ofimages to generate a prediction image. Accordingly, the image encoder1153 can generate a new prediction image closer to the original image.Hence, only a small amount of code to be assigned to residuals can berequired, thus making it possible to increase encoding efficiency.

Note that, at this time, the mobile phone 1100 simultaneously performs,using the audio codec 1159, analog-to-digital conversion on the audiocollected by the microphone (mic) 1121 during the capture of an imageusing the CCD camera 1116, and further encodes it.

The mobile phone 1100 multiplexes, using the multiplexing/demultiplexingunit 1157, the encoded image data supplied from the image encoder 1153and the digital audio data supplied from the audio codec 1159 using acertain scheme. The mobile phone 1100 performs, using themodulation/demodulation circuit unit 1158, a spread spectrum process onmultiplexed data obtained as a result, and performs, using thetransmission/reception circuit unit 1163, a digital-to-analog conversionprocess and a frequency conversion process. The mobile phone 1100transmits a transmission signal obtained by the conversion process to abase station that is not illustrated in the figure via the antenna 1114.The transmission signal (image data) transmitted to the base station issupplied to the other end of the communication via a network or thelike.

Note that in a case where no image data is to be transmitted, the mobilephone 1100 may also cause image data generated using the CCD camera 1116to be displayed on the liquid crystal display 1118 via the LCD controlunit 1155 without intervention of the image encoder 1153.

Furthermore, for example, in a case where data of a moving image filehaving a link to a simplified homepage or the like is to be received inthe data communication mode, the mobile phone 1100 receives, using thetransmission/reception circuit unit 1163 via the antenna 1114, a signaltransmitted from a base station, amplifies it, and further performs afrequency conversion process and an analog-to-digital conversionprocess. The mobile phone 1100 performs an inverse spread spectrumprocess on the received signal using the modulation/demodulation circuitunit 1158 to restore the original multiplexed data. The mobile phone1100 demultiplexes, using the multiplexing/demultiplexing unit 1157, themultiplexed data to separate it into encoded image data and audio data.

The mobile phone 1100 decodes encoded image data using the image decoder1156 to generate reproduction moving image data, and displays thereproduction moving image data on the liquid crystal display 1118 viathe LCD control unit 1155. This allows, for example, moving image dataincluded in a moving image file linked to the simplified homepage to bedisplayed on the liquid crystal display 1118.

The mobile phone 1100 uses the decoding apparatus 1 described above asthe image decoder 1156 that performs the above process. That is to say,similarly to the case of the decoding apparatus 1, the image decoder1156 obtains a motion prediction image from the frame (N−1) using motionvectors, and, in addition, obtains a spatial prediction image from theframe N through intra-prediction utilizing a decoded pixel value in theframe N. And the image decoder 1156 performs a filtering process on thetwo types of images to generate a prediction image. Accordingly, theimage decoder 1156 can generate a new prediction image closer to theoriginal image.

At this time, the mobile phone 1100 simultaneously converts digitalaudio data into an analog audio signal using the audio codec 1159, andcauses it to be output from the speaker 1117. This allows, for example,audio data included in the moving image file linked to the simplifiedhomepage to be reproduced.

Note that, similarly to the case of electronic mail, the mobile phone1100 may also be capable of causing received data linked to thesimplified homepage or the like to be recorded on (stored in) thestorage unit 1123 via the recording/reproducing unit 1162.

Furthermore, the mobile phone 1100 can also analyze, using the maincontrol unit 1150, a two-dimensional code obtained by the CCD camera1116 by capturing an image thereof, and acquire information recorded inthe two-dimensional code.

Moreover, the mobile phone 1100 can communicate with an external devicevia infrared light using an infrared communication unit 1181.

With the use of the encoding apparatus 101 as the image encoder 1153,the mobile phone 1100 can realize improvement in encoding efficiency bycompensating for the lack of prediction accuracy of intra-prediction byusing the prediction accuracy of inter-prediction when, for example,encoding image data generated by the CCD camera 1116 and transmittingthe encoded image data. Moreover, since the number of reference framesnecessary for inter-prediction can be reduced, the mobile phone 1100 canrealize a reduction in processing cost.

Furthermore, with the use of the decoding apparatus 1 as the imagedecoder 1156, the mobile phone 1100 can be configured to generate aprediction image by adding an image representing high-frequencycomponents to the intra-prediction image IP during decoding when, forexample, receiving data of a moving image file (encoded data) linked toa simplified homepage.

Thereby, the mobile phone 1100 can realize improvement in encodingefficiency by compensating for the lack of prediction ofintra-prediction accuracy by using the prediction accuracy ofinter-prediction. Moreover, the reduction in processing cost can also berealized by reducing the number of reference planes necessary forinter-prediction.

Note that while it has been explained that the mobile phone 1100 usesthe CCD camera 1116. However, the mobile phone 1100 may use, in place ofthe CCD camera 1116, an image sensor (CMOS image sensor) that uses aCMOS (Complementary Metal Oxide Semiconductor). Also in this case,similarly to the case of using the CCD camera 1116, the mobile phone1100 can capture an image of an object and generate image data of theimage of the object.

Furthermore, while the foregoing explanation has been given in thecontext of the mobile phone 1100, the decoding apparatus 1 and theencoding apparatus 101 can be applied to, similarly to the case of themobile phone 1100, for example, any apparatus having an image capturefunction or communication function similar to that of the mobile phone1100, such as a PDA (Personal Digital Assistants), a smartphone, a UMPC(Ultra Mobile Personal Computer), a netbook, or a notebook personalcomputer.

FIG. 30 is a block diagram illustrating an example configuration of amain part of a hard disk recorder that uses the decoding apparatus 1 andthe encoding apparatus 101 to which the present invention is applied.

A hard disk recorder (HDD recorder) 1200 illustrated in FIG. 30 is anapparatus that saves, in its built-in hard disk, audio data and videodata of a broadcast program included in a broadcast wave signal(television signal) transmitted from a satellite, a terrestrial antenna,or the like, which has been received by a tuner, and that provides auser with the saved data at a timing according to a user instruction.

The hard disk recorder 1200 can extract, for example, audio data andvideo data from a broadcast wave signal, decode them as appropriate, andstore them in the built-in hard disk. Furthermore, the hard diskrecorder 1200 can also acquire audio data or video data from anotherapparatus via, for example, a network, decode it as appropriate, andstore the decoded data in the built-in hard disk.

Further, the hard disk recorder 1200 can decode audio data and videodata recorded on, for example, the built-in hard disk, supply them to amonitor 1260, display an image thereof on a screen of the monitor 1260,and output audio thereof from a speaker of the monitor 1260.Furthermore, the hard disk recorder 1200 can also decode, for example,audio data and video data extracted from a broadcast wave signalacquired via a tuner or audio data and video data acquired from anotherapparatus via a network, supply them to the monitor 1260, display animage thereof on the screen of the monitor 1260, and output audiothereof from the speaker of the monitor 1260.

Of course, other operations are also possible.

As illustrated in FIG. 30, the hard disk recorder 1200 includes areceiving unit 1221, a demodulation unit 1222, a demultiplexer 1223, anaudio decoder 1224, a video decoder 1225, and a recorder control unit1226. The hard disk recorder 1200 further includes an EPG data memory1227, a program memory 1228, a work memory 1229, a display converter1230, an OSD (On Screen Display) control unit 1231, a display controlunit 1232, a recording/reproducing unit 1233, a D/A converter 1234, anda communication unit 1235.

Furthermore, the display converter 1230 includes a video encoder 1241.The recording/reproducing unit 1233 includes an encoder 1251 and adecoder 1252.

The receiving unit 1221 receives an infrared signal from a remotecontroller (not illustrated), converts it into an electrical signal, andoutputs it to the recorder control unit 1226. The recorder control unit1226 is composed of, for example, a microprocessor or the like, andexecutes various processes in accordance with a program stored in theprogram memory 1228. At this time, the recorder control unit 1226 usesthe work memory 1229 as necessary.

The communication unit 1235 is connected to a network, and performs acommunication process with another apparatus via the network. Forexample, the communication unit 1235 is controlled by the recordercontrol unit 1226 to communicate with a tuner (not illustrated) and tooutput mainly a channel selection control signal to the tuner.

The demodulation unit 1222 demodulates a signal supplied from the tuner,and outputs it to the demultiplexer 1223. The demultiplexer 1223demultiplexes the data supplied from the demodulation unit 1222 intoaudio data, video data, and EPG data, and outputs them to the audiodecoder 1224, the video decoder 1225, and the recorder control unit1226, respectively.

The audio decoder 1224 decodes the input audio data and outputs thedecoded audio data to the recording/reproducing unit 1233. The videodecoder 1225 decodes the input video data and outputs the decoded videodata to the display converter 1230. The recorder control unit 1226supplies the input EPG data to the EPG data memory 1227 to store the EPGdata.

The display converter 1230 encodes, using the video encoder 1241, videodata supplied from the video decoder 1225 or the recorder control unit1226 into video data of, for example, the NTSC (National TelevisionStandards Committee) scheme, and outputs it to the recording/reproducingunit 1233. Furthermore, the display converter 1230 converts the screensize of the video data supplied from the video decoder 1225 or therecorder control unit 1226 into the size corresponding to the size ofthe monitor 1260, converts the video data into video data of the NTSCscheme using the video encoder 1241, converts it into an analog signal,and outputs it to the display control unit 1232.

Under the control of the recorder control unit 1226, the display controlunit 1232 superimposes an OSD signal output from the OSD (On ScreenDisplay) control unit 1231 onto the video signal input from the displayconverter 1230, and outputs it to the display of the monitor 1260 todisplay it.

The audio data output from the audio decoder 1224, which has beenconverted into an analog signal by the D/A converter 1234, is alsosupplied to the monitor 1260. The monitor 1260 outputs the audio signalfrom its built-in speaker.

The recording/reproducing unit 1233 includes a hard disk as a storagemedium on which video data, audio data, and the like are recorded.

The recording/reproducing unit 1233 encodes, using the encoder 1251, forexample, the audio data supplied from the audio decoder 1224.Furthermore, the recording/reproducing unit 1233 encodes, using theencoder 1251, the video data supplied from the video encoder 1241 of thedisplay converter 1230. The recording/reproducing unit 1233 combines theencoded data of the audio data and the encoded data of the video datausing a multiplexer. The recording/reproducing unit 1233 performschannel coding on resulting composite data, amplifies it, and writes thedata to the hard disk via a recording head.

The recording/reproducing unit 1233 reproduces the data recorded on thehard disk via a reproduction head, amplifies it, and separates it intoaudio data and video data using a demultiplexer. Therecording/reproducing unit 1233 decodes the audio data and the videodata using the decoder 1252. The recording/reproducing unit 1233performs D/A conversion on the decoded audio data, and outputs it to thespeaker of the monitor 1260. Furthermore, the recording/reproducing unit1233 performs D/A conversion on the decoded video data, and outputs itto the display of the monitor 1260.

The recorder control unit 1226 reads the latest EPG data from the EPGdata memory 1227 on the basis of a user instruction indicated by theinfrared signal from the remote controller which is received via thereceiving unit 1221, and supplies the EPG data to the OSD control unit1231. The OSD control unit 1231 produces image data corresponding to theinput EPG data, and outputs it to the display control unit 1232. Thedisplay control unit 1232 outputs the video data input from the OSDcontrol unit 1231 to the display of the monitor 1260 to display it. Thisallows an EPG (electronic program guide) to be displayed on the displayof the monitor 1260.

Furthermore, the hard disk recorder 1200 can also acquire various datasuch as video data, audio data, and EPG data supplied from anotherapparatus via a network such as the Internet.

The communication unit 1235 is controlled by the recorder control unit1226 to acquire encoded data such as video data, audio data, and EPGdata transmitted from another apparatus via a network, and supplies itto the recorder control unit 1226. The recorder control unit 1226supplies, for example, encoded data of the acquired video data and audiodata to the recording/reproducing unit 1233 to store it in a hard disk.At this time, the recorder control unit 1226 and therecording/reproducing unit 1233 may perform a process such asre-encoding as necessary.

Furthermore, the recorder control unit 1226 decodes encoded data of theacquired video data and audio data, and supplies obtained video data tothe display converter 1230. The display converter 1230 processes thevideo data supplied from the recorder control unit 1226 in a mannersimilar to that of the video data supplied from the video decoder 1225,and supplies the resulting video data to the monitor 1260 via thedisplay control unit 1232 to display an image thereof.

Furthermore, along with the display of the image, the recorder controlunit 1226 may supply decoded audio data to the monitor 1260 via the D/Aconverter 1234 and output audio thereof from the speaker.

Further, the recorder control unit 1226 decodes encoded data of theacquired EPG data, and supplies the decoded EPG data to the EPG datamemory 1227.

The hard disk recorder 1200 as above uses the decoding apparatus 1 asthe video decoder 1225, the decoder 1252, and the built-in decoder ofthe recorder control unit 1226. That is to say, similarly to the case ofthe decoding apparatus 1, the video decoder 1225, the decoder 1252, andthe built-in decoder of the recorder control unit 1226 obtain a motionprediction image from the frame (N−1) using motion vectors, and, inaddition, obtain a spatial prediction image from the frame N throughintra-prediction utilizing a decoded pixel value in the frame N. And thevideo decoder 1225, the decoder 1252, and the built-in decoder of therecorder control unit 1226 perform a filtering process on the two typesof images to generate a prediction image. Accordingly, the video decoder1225, the decoder 1252, and the built-in decoder of the recorder controlunit 1226 can generate a new prediction image closer to the originalimage.

Accordingly, the hard disk recorder 1200 can be configured to generate aprediction image by adding an image representing high-frequencycomponents to the intra-prediction image IP during decoding when, forexample, receiving video data (encoded data) by a tuner or by thecommunication unit 1235 or when reproducing video data (encoded) from ahard disk by the recording/reproducing unit 1233.

Thereby, the hard disk recorder 1200 can realize improvement in encodingefficiency by compensating for the lack of prediction accuracy ofintra-prediction by using the prediction accuracy of inter-prediction.Moreover, the reduction in processing cost can also be realized byreducing the number of reference planes necessary for inter-prediction.

Furthermore, hard disk recorder 1200 uses the encoding apparatus 101 asthe encoder 1251. Similarly to the case of the encoding apparatus 101,the encoder 1251 performs a filtering process on a motion predictionimage and an intra-prediction image using motion vectors to generate aprediction image. Accordingly, the encoder 1251 can generate a newprediction image closer to the original image. Hence, only a smallamount of code to be assigned to residuals can be required, thus makingit possible to increase encoding efficiency.

Accordingly, the hard disk recorder 1200 can improve encoding efficiencyby compensating for the lack of prediction accuracy of intra-predictionby using the prediction accuracy of inter-prediction when, for example,recording encoded data on a hard disk. Moreover, since the number ofreference frames necessary for inter-prediction, the hard disk recorder1200 can realize a reduction in processing cost.

Note that while the hard disk recorder 1200 that records video data andaudio data on a hard disk has been explained, of course, any type ofrecording medium may be used. For example, even a recorder that uses arecording medium other than a hard disk, such as a flash memory, anoptical disk, or a videotape, can also use the decoding apparatus 1 andthe encoding apparatus 101 in a manner similar to that in the case ofthe hard disk recorder 1200 described above.

FIG. 31 is a block diagram illustrating an example configuration of amain part of a camera that uses the decoding apparatus 1 and theencoding apparatus 101 to which the present invention is applied.

A camera 1300 illustrated in FIG. 31 captures an image of an object, andcauses the image of the object to be displayed on an LCD 1316 or recordsit on a recording medium 1333 as image data.

A lens block 1311 causes light (that is, video of an object) to beincident on a CCD/CMOS 1312. The CCD/CMOS 1312 is an image sensor thatuses a CCD or a CMOS, converts the intensity of received light into anelectrical signal, and supplies it to a camera signal processing unit1313.

The camera signal processing unit 1313 converts the electrical signalsupplied from the CCD/CMOS 1312 into Y, Cr, and Cb color differencesignals, and supplies them to an image signal processing unit 1314.Under the control of a controller 1321, the image signal processing unit1314 performs certain image processing on the image signal supplied fromthe camera signal processing unit 1313, or encodes the image signalusing an encoder 1341. The image signal processing unit 1314 suppliesencoded data generated by encoding the image signal to the decoder 1315.Further, the image signal processing unit 1314 acquires display datagenerated by an on-screen display (OSD) 1320, and supplies it to adecoder 1315.

In the above processes, the camera signal processing unit 1313 utilizesa DRAM (Dynamic Random Access Memory) 1318 connected via a bus 1317, andcauses image data, encoded data obtained by encoding the image data, orthe like to be held in the DRAM 1318 as necessary.

The decoder 1315 decodes the encoded data supplied from the image signalprocessing unit 1314, and supplies obtained image data (decoded imagedata) to the LCD 1316. Furthermore, the decoder 1315 supplies thedisplay data supplied from the image signal processing unit 1314 to theLCD 1316. The LCD 1316 combines the image of the decoded image datasupplied from the decoder 1315 and the image of the display data, asappropriate, and displays a resulting composite image.

Under the control of the controller 1321, the on-screen display 1320outputs display data such as a menu screen formed of signs, characters,or figures, and an icon to the image signal processing unit 1314 via thebus 1317.

The controller 1321 executes various processes on the basis of a signalindicating the content of a command issued by a user using an operationunit 1322, and also controls the image signal processing unit 1314, theDRAM 1318, an external interface 1319, the on-screen display 1320, amedium drive 1323, and the like via the bus 1317. A FLASH ROM 1324stores programs, data, and the like necessary for the controller 1321 toexecute various processes.

For example, the controller 1321 can encode image data stored in theDRAM 1318 or decode encoded data stored in the DRAM 1318 on behalf ofthe image signal processing unit 1314 or the decoder 1315. At this time,the controller 1321 may perform an encoding or decoding process using ascheme similar to the encoding or decoding scheme of the image signalprocessing unit 1314 or the decoder 1315, or may perform an encoding ordecoding process using a scheme that is not supported by the imagesignal processing unit 1314 or the decoder 1315.

Furthermore, for example, in a case where an instruction for startingprinting an image has been issued from the operation unit 1322, thecontroller 1321 reads image data from the DRAM 1318, and supplies it toa printer 1334 connected to the external interface 1319 via the bus 1317to print it.

Further, for example, in a case where an instruction for recording animage has been issued from the operation unit 1322, the controller 1321reads encoded data from the DRAM 1318, and supplies it to the recordingmedium 1333 attached to the medium drive 1323 via the bus 1317 to storeit.

The recording medium 1333 is, for example, any readable and rewritableremovable medium such as a magnetic disk, a magneto-optical disk, anoptical disk, or a semiconductor memory. The recording medium 1333 mayalso be, of course, any type of removable medium, and may be a tapedevice, a disk, or a memory card. Of course, a non-contact IC card orthe like may also be used.

Furthermore, the medium drive 1323 and the recording medium 1333 mayalso be integrally formed into, for example, a non-portable storagemedium such as a built-in hard disk drive or an SSD (Solid State Drive).

The external interface 1319 is composed of, for example, a USBinput/output terminal or the like, and is connected to the printer 1334in a case where an image is to be printed. Furthermore, a drive 1331 isconnected to the external interface 1319 as necessary, and a removablemedium 1332 such as a magnetic disk, an optical disk, or amagneto-optical disk is attached as appropriate, so that a computerprogram read therefrom is installed into the FLASH ROM 1324 asnecessary.

Further, the external interface 1319 includes a network interface thatis connected to a certain network such as a LAN or the Internet. Thecontroller 1321 can read encoded data from the DRAM 1318 in accordancewith an instruction from, for example, the operation unit 1322, andsupply it to another apparatus connected via a network from the externalinterface 1319. Furthermore, the controller 1321 can acquire, via theexternal interface 1319, encoded data or image data supplied fromanother apparatus via a network, and hold it in the DRAM 1318 or supplyit to the image signal processing unit 1314.

The camera 1300 as above uses the decoding apparatus 1 as the decoder1315. That is to say, similarly to the case of the decoding apparatus 1,the decoder 1315 obtains a motion prediction image from the frame (N−1)using motion vectors, and, in addition, obtains a spatial predictionimage from the frame N through intra-prediction utilizing a decodedpixel value in the frame N. And the decoder 1315 performs a filteringprocess on the two types of images to generate a prediction image.Accordingly, the decoder 1315 can generate a new prediction image closerto the original image.

Accordingly, the camera 1300 can be configured to generate a predictionimage by adding an image representing high-frequency components to theintra-prediction image IP during decoding for, for example, image datagenerated by the CCD/CMOS 1312, when reading encoded data of video datafrom the DRAM 1318 or the recording medium 1333, or when acquiringencoded data of video data via a network.

Furthermore, the camera 1300 uses the encoding apparatus 101 as theencoder 1341. Similarly to the case of the encoding apparatus 101, theencoder 1341 performs a filtering process on a motion prediction imageand an intra-prediction image using motion vectors to generate aprediction image. Accordingly, the encoder 1341 can generate a newprediction image closer to the original image. Hence, only a smallamount of code to be assigned to residuals can be required, thus makingit possible to increase encoding efficiency.

Accordingly, the camera 1300 can improve encoding efficiency bycompensating for the lack of prediction accuracy of intra-prediction byusing the prediction accuracy of inter-prediction when, for example,recording encoded data on the DRAM 1318 or the recording medium 1333 orwhen providing encoded data to another apparatus. Moreover, since thenumber of reference frames necessary for inter-prediction can bereduced, the camera 1300 can realize a reduction in processing cost.

Note that a decoding method of the decoding apparatus 1 may be appliedto a decoding process performed by the controller 1321. Similarly, anencoding method of the encoding apparatus 101 may be applied to anencoding process performed by the controller 1321.

Furthermore, image data captured by the camera 1300 may be that of amoving image or a still image.

Of course, the decoding apparatus 1 and the encoding apparatus 101 canalso be applied to an apparatus or a system other than the apparatusesdescribed above.

Furthermore, the dimensions of a macroblock are arbitrary. The presentinvention can be applied to, for example, a macroblock having anydimension as illustrated in FIG. 32. For example, the present inventioncan be applied not only to a normal macroblock of 16×16 pixels but alsoto an extended macroblock (extension macroblock) such as a macroblock of32×32 pixels.

In FIG. 32, in the upper portion, macroblocks composed of 32×32 pixelsthat are divided by blocks (partitions) of 32×32 pixels, 32×16 pixels,16×32 pixels, and 16×16 pixels are illustrated in sequence from theleft. Furthermore, in the middle portion, blocks composed of 16×16pixels that are divided by blocks of 16×16 pixels, 16×8 pixels, 8×16pixels, and 8×8 pixels are illustrated in sequence from the left.Moreover, in the lower portion, blocks composed of 8×8 pixels that aredivided by blocks of 8×8 pixels, 8×4 pixels, 4×8 pixels, and 4×4 pixelsare illustrated in sequence from the left.

That is, macroblocks of 32×32 pixels can be processed by blocks of 32×32pixels, 32×16 pixels, 16×32 pixels, and 16×16 pixels illustrated in theupper portion.

The block of 16×16 pixels illustrated on the right side in the upperportion can be processed by, similarly to the H.264/AVC scheme, blocksof 16×16 pixels, 16×8 pixels, 8×16 pixels, and 8×8 pixels illustrated inthe middle portion.

The block of 8×8 pixels illustrated on the right side in the middleportion can be processed by, similarly to the H.264/AVC scheme, blocksof 8×8 pixels, 8×4 pixels, 4×8 pixels, and 4×4 pixels illustrated in thelower portion.

The above blocks can be classified into the following three hierarchicallayers. That is, the blocks of 32×32 pixels, 32×16 pixels, and 16×32pixels illustrated in the upper portion in FIG. 32 are referred to asblocks in a first hierarchical layer. The block of 16×16 pixelsillustrated on the right side in the upper portion, and the blocks of16×16 pixels, 16×8 pixels, and 8×16 pixels illustrated in the middleportion are referred to as blocks in a second hierarchical layer. Theblock of 8×8 pixels illustrated on the right side in the middle portion,and the blocks of 8×8 pixels, 8×4 pixels, 4×8 pixels, and 4×4 pixelsillustrated in the lower portion are referred to as blocks in a thirdhierarchical layer.

With the adoption of the above hierarchical layer structure, with regardto the blocks equal to or lower than the block of 16×16 pixels, largerblocks can be defined as supersets thereof while maintaining thecompatibility with the H.264/AVC scheme.

For example, the decoding apparatus 1 and the encoding apparatus 101 maybe configured to generate a prediction image for each hierarchicallayer. Furthermore, for example, the decoding apparatus 1 and theencoding apparatus 101 may be configured to utilize a prediction image,which is generated in the first hierarchical layer that has a largerblock size than the second hierarchical layer, also for the secondhierarchical layer.

Macroblocks for which encoding is to be performed using a comparativelylarge block size, such as the first hierarchical layer and the secondhierarchical layer, do not include comparatively high-frequencycomponents. In contrast, macroblocks for which encoding is to beperformed using a comparatively small block size, such as the thirdhierarchical layer, are considered to include comparativelyhigh-frequency components.

Thus, prediction images are separately generated in accordance withindividual hierarchical layers having different block sizes, thus makingfeasible improvement in encoding performance suitable for localcharacteristics of an image.

REFERENCE SIGNS LIST

-   -   1 decoding apparatus, 21 motion prediction/compensation circuit,        41 prediction mode determination circuit, 42 unidirectional        prediction circuit, 43 bidirectional prediction circuit, 44        prediction circuit, 45 filtering circuit, 51 motion compensation        circuit, 52 intra-prediction circuit, 61 difference calculation        circuit, 62 low-pass filter circuit, 63 gain adjustment circuit,        64 high-pass filter circuit, 65 gain adjustment circuit, 66        adder circuit, 67 adder circuit

1. (canceled)
 2. An image processing system comprising: circuitry configured to decode an encoded image; add together an image that has been decoded and a prediction image, and generate a decoded image; perform motion compensation using, as a reference frame, a frame formed of the decoded image and using a motion vector in the encoded image, and extract a motion compensation image corresponding to the prediction image from the reference frame; perform intra-frame prediction for a present frame for which the prediction image is to be generated, and generate an intra-prediction image corresponding to the prediction image from a portion of the decoded image; and generate the prediction image by performing a filtering process to compensate the extracted motion compensation image and the generated intra-prediction image for high-frequency components by utilizing correlation in a time direction that is included in the extracted motion compensation image and the generated intra-prediction image, said generation of the prediction image including: applying a first filter to a difference image between the extracted motion compensation image and the generated intra-prediction image, and applying a second filter, different from the first filter, to an image obtained by applying the first filter.
 3. The image processing system according to claim 2, wherein the first filter is a low-pass filter and the second filter is a high-pass filter.
 4. The image processing system according to claim 2, wherein said generation of the prediction image further includes adding the image obtained by applying the first filter and an image obtained by applying the second filter to the generated intra-prediction image.
 5. The image processing system according to claim 4, wherein the first filter is a low-pass filter and the second filter is a high-pass filter.
 6. The image processing system according to claim 2, wherein the circuitry is configured to add the image obtained by applying the first filter and the image obtained by applying the second filter to a motion compensation image extracted from a preceding frame with respect to a time of the prediction image.
 7. The image processing system according to claim 6, wherein the first filter is a low-pass filter and the second filter is a high-pass filter.
 8. The image processing system according to claim 2, wherein the circuitry is configured to: perform unidirectional prediction using a plurality of motion compensation images, and generate the prediction image; perform bidirectional prediction using a plurality of motion compensation images, and generate the prediction image; and judge whether the prediction image is to be generated through unidirectional prediction, generated through bidirectional prediction, or generated through the filtering process using an identification flag included in a header of the encoded image.
 9. An image processing method comprising: decoding, using a processor, an encoded image; adding together, using the processor, an image that has been decoded and a prediction image, and generating a decoded image; performing, using the processor, motion compensation using, as a reference frame, a frame formed of the generated decoded image and using a motion vector in the encoded image, and extracting a motion compensation image corresponding to the prediction image from the reference frame; performing, using the processor, intra-frame prediction for a present frame for which the prediction image is to be generated, and generating an intra-prediction image corresponding to the prediction image from a portion of the decoded image; and generating, using the processor, the prediction image by performing a filtering process for compensating the motion compensation image and the intra-prediction image for high-frequency components by utilizing correlation in a time direction that is included in the motion compensation image and the intra-prediction image, wherein said generating the prediction image includes: performing a first filtering of a difference image between the extracted motion compensation image and the generated intra-prediction image, and performing a second filtering, different from the first filtering, to an image obtained by performing the first filtering.
 10. The image processing method according to claim 9, wherein the first filtering is low-pass filtering and the second filtering is high-pass filtering.
 11. The image processing method according to claim 9, wherein said generating the prediction image further includes adding the image obtained by performing the first filtering and an image obtained by performing the second filtering to the generated intra-prediction image.
 12. The image processing method according to claim 11, wherein the first filtering is low-pass filtering and the second filtering is high-pass filtering.
 13. The image processing method according to claim 9, further comprising adding, using the processor, the image obtained by performing the first filtering and the image obtained by applying the second filtering to a motion compensation image extracted from a preceding frame with respect to a time of the prediction image.
 14. The image processing method according to claim 13, wherein the first filtering is low-pass filtering and the second filtering is high-pass filtering.
 15. The image processing method according to claim 9, further comprising: performing, using the processor, unidirectional prediction using a plurality of motion compensation images, and generating the prediction image; performing, using the processor, bidirectional prediction using a plurality of motion compensation images, and generate the prediction image; and judging, using the processor, whether the prediction image is to be generated through unidirectional prediction, generated through bidirectional prediction, or generated through the filtering process using an identification flag included in a header of the encoded image. 